Skip to content

enhancement: implement getContext method for device emulation #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tusharmctrl wants to merge 8 commits into
base: main
Choose a base branch
from
Open

enhancement: implement getContext method for device emulation #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
09f999a
Add support for running automation for mobile devices to check UI/UX
tusharmctrl Apr 30, 2025
b64214e
Added missing property in a type
tusharmctrl Apr 30, 2025
4ffaa46
feat: add device support in localConfig
tusharmctrl May 1, 2025
fb498f8
fix: remove unwanted type
tusharmctrl May 1, 2025
0f6ec53
remove unwanted code from HyperbrowserProvider
tusharmctrl May 1, 2025
e066278
final cleanup of useMobile
tusharmctrl May 1, 2025
30f4cb2
feat: added new param
tusharmctrl May 7, 2025
436f362
fix: resolved sharp error about deviceFactor
tusharmctrl May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions src/agent/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ export class HyperAgent<T extends BrowserProviders = "Local"> {
: LocalBrowserProvider;
private browserProviderType: T;
private actions: Array<AgentActionDefinition> = [...DEFAULT_ACTIONS];
private clientType: "desktop" | "mobile";
Copy link
Contributor

@AHarmlessPyro AHarmlessPyro May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you take the types directly from the devices offered from the devices list ?
Something like private clientType?: keyof typeof devices;

Copy link
Contributor

@AHarmlessPyro AHarmlessPyro May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the constructor too can you assign the clientType on the object instance ?
Like this.clientType = params.clientType

Copy link
Author

@tusharmctrl tusharmctrl May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - I tried this, but it wouldn't give actual suggestions to user - reason being, playwright doesnt do export on Devices type. So, considering user perspective - asking user to select mobile / desktop - would make more sense since we're not able to provide suggestions.


public browser: Browser | null = null;
public context: BrowserContext | null = null;
Expand Down Expand Up @@ -76,6 +77,7 @@ export class HyperAgent<T extends BrowserProviders = "Local"> {
this.llm = params.llm;
}
this.browserProviderType = (params.browserProvider ?? "Local") as T;
this.clientType = params.clientType ?? "desktop";

this.browserProvider = (
this.browserProviderType === "Hyperbrowser"
Expand All @@ -100,9 +102,13 @@ export class HyperAgent<T extends BrowserProviders = "Local"> {
public async initBrowser(): Promise<Browser> {
if (!this.browser) {
this.browser = await this.browserProvider.start();
this.context = await this.browser.newContext({
viewport: null,
});
this.context = await this.browserProvider.getContext(this.clientType);

if (!this.context) {
this.context = await this.browser.newContext({
viewport: null,
});
}

// Inject script to track event listeners
await this.context.addInitScript(() => {
Expand Down
27 changes: 26 additions & 1 deletion src/browser-providers/hyperbrowser.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
import { chromium, Browser, ConnectOverCDPOptions } from "playwright";
import {
chromium,
Browser,
ConnectOverCDPOptions,
BrowserContext,
devices,
} from "playwright";
import { Hyperbrowser } from "@hyperbrowser/sdk";
import {
CreateSessionParams,
Expand Down Expand Up @@ -62,6 +68,25 @@ export class HyperbrowserProvider extends BrowserProvider<SessionDetail> {
}
}

public async getContext(
device: string = "desktop"
): Promise<BrowserContext | null> {
if (!this.browser) return null;

if (device === "mobile") {
const iPhone = devices["iPhone 12"];
return await this.browser.newContext({
userAgent: iPhone.userAgent,
viewport: {
width: iPhone.viewport.width + 50,
height: iPhone.viewport.height + 50,
},
});
}

return await this.browser.newContext();
}

public getSession() {
if (!this.session) {
return null;
Expand Down
31 changes: 30 additions & 1 deletion src/browser-providers/local.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
import { chromium, Browser, LaunchOptions } from "playwright";
import {
chromium,
Browser,
LaunchOptions,
devices,
BrowserContext,
} from "playwright";
import BrowserProvider from "@/types/browser-providers/types";

export class LocalBrowserProvider extends BrowserProvider<Browser> {
options: Omit<Omit<LaunchOptions, "headless">, "channel"> | undefined;
session: Browser | undefined;

constructor(options?: Omit<Omit<LaunchOptions, "headless">, "channel">) {
super();
this.options = options;
}

async start(): Promise<Browser> {
const launchArgs = this.options?.args ?? [];
const browser = await chromium.launch({
Expand All @@ -19,13 +27,34 @@ export class LocalBrowserProvider extends BrowserProvider<Browser> {
this.session = browser;
return this.session;
}

async close(): Promise<void> {
return await this.session?.close();
}

public getSession() {
if (!this.session) {
return null;
}
return this.session;
}

public async getContext(
Copy link
Contributor

@AHarmlessPyro AHarmlessPyro May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add this method on the abstract base class that they inherit from ?

device: string = "desktop"
): Promise<BrowserContext | null> {
if (!this.session) return null;

if (device === "mobile") {
const iPhone = devices["iPhone 12"];
return await this.session.newContext({
Copy link
Contributor

@AHarmlessPyro AHarmlessPyro May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass in the entire device object, don't specify individual params here

 this.session.newContext({...devices[device]})

Copy link
Author

@tusharmctrl tusharmctrl May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that - that wouldn't work because, there's this property where sharp is contradicting.

When we pass the entire object, it will throw an error like:

Error: Image to composite must have same dimensions or smaller
    at Sharp.toBuffer (/home/tushar-rupani/missionctrl/HyperAgent/node_modules/sharp/lib/output.js:163:17)
    at compositeScreenshot (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/tools/agent.ts:38:6)
    at runAgentTask (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/tools/agent.ts:156:31)
    at HyperAgent.executeTask (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/index.ts:335:14)
    at <anonymous> (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/test.ts:13:18)

Copy link
Author

@tusharmctrl tusharmctrl May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason behind that is deviceScaleFactor

Copy link
Author

@tusharmctrl tusharmctrl May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And isMobile too.

userAgent: iPhone.userAgent,
viewport: {
width: iPhone.viewport.width + 50,
height: iPhone.viewport.height + 50,
},
});
}

return await this.session.newContext();
}
}
5 changes: 3 additions & 2 deletions src/types/browser-providers/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import { Browser } from "playwright";
import { Browser, BrowserContext } from "playwright";

abstract class BrowserProvider<T> {
abstract session: unknown;
abstract start(): Promise<Browser>;
abstract close(): Promise<void>;
abstract getSession(): T|null;
abstract getSession(): T | null;
abstract getContext(device?: string): Promise<BrowserContext | null>;
}

export default BrowserProvider;
2 changes: 1 addition & 1 deletion src/types/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ export interface HyperAgentConfig<T extends BrowserProviders = "Local"> {

debug?: boolean;
llm?: BaseChatModel;

clientType?: "mobile" | "desktop";
hyperbrowserConfig?: Omit<
NonNullable<ConstructorParameters<typeof HyperbrowserProvider>[0]>,
"debug"
Expand Down