Conversation
Member
GJFR
commented
Nov 28, 2024
GJFR
commented
- New experiment functionality:
- Dynamic python back-end scripting
- User interaction simulation
- All requests received during an experiment are stored
- New UI functionality:
- Removing data points by SHIFT clicking
- New experiment creation options
- Various improvements and bug fixes
* Fix code highlighting modes and add new .py filetype * Add empty testing Python server experiment * Fix Python ace mode * Fix loading Python files * Simple Python server fetch experiment * Basic python directory proxying * Dynamically import and execute Python experiment files * Pass request data to custom Python files * Default Python server template content * Add JSON highlighting * Remove python server headers * Disable creating __pycache__ folder with dynamic imports
- Change internal tree representation for experiments - Store python backend logic as files without folder - Existing url_queue.txt files can be editor through web UI
After trying out the devtools protocols in BugHog, the two simple
implementations I had prepared worked only for Chromium 70+ and Firefox
127+. Even though there were some other protocols implemented in
previous browser versions, it would require a lot of effort to implement
all of these and correctly match them to the supported browser versions.
Further, there would likely be no way to target versions approx. 20-50.
As a result, I decided to use PyAutoGUI as you suggested. It works in
all browsers and versions and for practical usage it seems almost as
good as the devtools protocols.
Instead of `url_queue.txt`, users can choose to provide the experiment
configuration in `interaction_script.cmd`. These are the commands
implemented so far:
- `NAVIGATE url` Terminates the previous browser window and opens a new
browser window on the specified URL. Further, it waits some time (1 sec
for Chromium, 2 secs for Firefox) for the page to load.
- `CLICK_POSITION x y` Clicks on specific coordinates on the screen (not
necessarily the browser window), The argument values can be absolute in
pixels, percentage of the screen, or a combination of both - e.g.,
`CLICK_POSITION 100 50%` clicks 100px from the left screen border and
50% of the screen height.
- `CLICK element_id` Clicks on an element with the specified ID.
Currently, the ID can be one of `one`, `two`, `three`, `four`, `five`,
`six`. This is because PyAutoGUI can search for the location of a visual
match on the screen. Therefore, I prepared styles in `res/bughog.css`
that style elements with these IDs to boxes of distinct colors. This
allows us to bypass the limitation of having to know the exact screen
coordinates of an element.
- `WRITE text` Types the text into the focused element.
- `PRESS key`
- `HOLD key`
- `RELEASE key`
- `HOTKEY key1 key2 ...` A combination of `HOLD` and `RELEASE` for
multiple keys. E.g., `HOTKEY ctrl c`.
- `SLEEP seconds` Usually should not be necessary to use because
navigation implicitly includes sleeping.
- `SCREENSHOT file_name` Captures the screen and stores the result in
`logs/screenshots/{PROJECT}-{EXPERIMENT}-{file_name}-{BROWSER}-{VERSION}.jpg`.
Very useful for debugging
A simple experiment can be found in `Support/AutoGUI`. It got
successfully reproduced in all versions of both browsers.
We can possibly implement some browser-specific behaviour as well, e.g.,
bookmarking a string where the script would include only `BOOKMARK text`
and based on the browser version, the correct shortcut would be pressed
and screen positions clicked.
Besides this, I made the following changes:
- Extracted the default file templates to separate files and added
templates for all file types
- Implemented adding and modifying `interaction_script.cmd` and
`url_queue.txt` from the web UI
- Implemented a custom highlighting mode for `interaction_script.cmd` in
the experiment editor
- Fixed loading resources from `/res/`
…ive and add favicon for experiment pages
Additionally, fix bug where cached Firefox binary data was misinterpreted.
Because BGB Search was not aware of states that were already in progress when BGB Sequence had finished (no database entry yet), some states would be evaluated more than once on occasion. This especially happened when there are more concurrent containers than states to evaluate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.