Skip to content

Adding dry run option to request, allows requests to be validated and for an estimated cost to be returned, prior to full generation #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
johnsabath wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Adding dry run option to request, allows requests to be validated and for an estimated cost to be returned, prior to full generation #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gooseai/dashboard/dashboard.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 1 addition & 7 deletions gooseai/engines/engines_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,7 @@

var jspb = require('google-protobuf');
var goog = jspb;
var global = (function() {
if (this) { return this; }
if (typeof window !== 'undefined') { return window; }
if (typeof global !== 'undefined') { return global; }
if (typeof self !== 'undefined') { return self; }
return Function('return this')();
}.call(null));
var global = (function() { return this || window || global || self || Function('return this')(); }).call(null);

goog.exportSymbol('proto.gooseai.EngineInfo', null, global);
goog.exportSymbol('proto.gooseai.EngineTokenizer', null, global);
Expand Down
338 changes: 181 additions & 157 deletions gooseai/generation/generation.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions gooseai/generation/generation_pb.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -838,6 +838,11 @@ export class Answer extends jspb.Message {
setArtifactsList(value: Array<Artifact>): void;
addArtifacts(value?: Artifact, index?: number): Artifact;

hasEstimatedCost(): boolean;
clearEstimatedCost(): void;
getEstimatedCost(): number;
setEstimatedCost(value: number): void;

serializeBinary(): Uint8Array;
toObject(includeInstance?: boolean): Answer.AsObject;
static toObject(includeInstance: boolean, msg: Answer): Answer.AsObject;
Expand All @@ -856,6 +861,7 @@ export namespace Answer {
created: number,
meta?: AnswerMeta.AsObject,
artifactsList: Array<Artifact.AsObject>,
estimatedCost: number,
}
}

Expand Down Expand Up @@ -894,6 +900,9 @@ export class Request extends jspb.Message {
getConditioner(): ConditionerParameters | undefined;
setConditioner(value?: ConditionerParameters): void;

getDryRun(): boolean;
setDryRun(value: boolean): void;

getParamsCase(): Request.ParamsCase;
serializeBinary(): Uint8Array;
toObject(includeInstance?: boolean): Request.AsObject;
Expand All @@ -915,6 +924,7 @@ export namespace Request {
classifier?: ClassifierParameters.AsObject,
asset?: AssetParameters.AsObject,
conditioner?: ConditionerParameters.AsObject,
dryRun: boolean,
}

export enum ParamsCase {
Expand Down
82 changes: 80 additions & 2 deletions gooseai/generation/generation_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6537,7 +6537,8 @@ proto.gooseai.Answer.toObject = function(includeInstance, msg) {
created: jspb.Message.getFieldWithDefault(msg, 4, 0),
meta: (f = msg.getMeta()) && proto.gooseai.AnswerMeta.toObject(includeInstance, f),
artifactsList: jspb.Message.toObjectList(msg.getArtifactsList(),
proto.gooseai.Artifact.toObject, includeInstance)
proto.gooseai.Artifact.toObject, includeInstance),
estimatedCost: jspb.Message.getFloatingPointFieldWithDefault(msg, 8, 0.0)
};

if (includeInstance) {
Expand Down Expand Up @@ -6600,6 +6601,10 @@ proto.gooseai.Answer.deserializeBinaryFromReader = function(msg, reader) {
reader.readMessage(value,proto.gooseai.Artifact.deserializeBinaryFromReader);
msg.addArtifacts(value);
break;
case 8:
var value = /** @type {number} */ (reader.readDouble());
msg.setEstimatedCost(value);
break;
default:
reader.skipField();
break;
Expand Down Expand Up @@ -6673,6 +6678,13 @@ proto.gooseai.Answer.serializeBinaryToWriter = function(message, writer) {
proto.gooseai.Artifact.serializeBinaryToWriter
);
}
f = /** @type {number} */ (jspb.Message.getField(message, 8));
if (f != null) {
writer.writeDouble(
8,
f
);
}
};


Expand Down Expand Up @@ -6823,6 +6835,42 @@ proto.gooseai.Answer.prototype.clearArtifactsList = function() {
};


/**
* optional double estimated_cost = 8;
* @return {number}
*/
proto.gooseai.Answer.prototype.getEstimatedCost = function() {
return /** @type {number} */ (jspb.Message.getFloatingPointFieldWithDefault(this, 8, 0.0));
};


/**
* @param {number} value
* @return {!proto.gooseai.Answer} returns this
*/
proto.gooseai.Answer.prototype.setEstimatedCost = function(value) {
return jspb.Message.setField(this, 8, value);
};


/**
* Clears the field making it undefined.
* @return {!proto.gooseai.Answer} returns this
*/
proto.gooseai.Answer.prototype.clearEstimatedCost = function() {
return jspb.Message.setField(this, 8, undefined);
};


/**
* Returns whether this field is set.
* @return {boolean}
*/
proto.gooseai.Answer.prototype.hasEstimatedCost = function() {
return jspb.Message.getField(this, 8) != null;
};



/**
* List of repeated fields within this message type.
Expand Down Expand Up @@ -6897,7 +6945,8 @@ proto.gooseai.Request.toObject = function(includeInstance, msg) {
image: (f = msg.getImage()) && proto.gooseai.ImageParameters.toObject(includeInstance, f),
classifier: (f = msg.getClassifier()) && proto.gooseai.ClassifierParameters.toObject(includeInstance, f),
asset: (f = msg.getAsset()) && proto.gooseai.AssetParameters.toObject(includeInstance, f),
conditioner: (f = msg.getConditioner()) && proto.gooseai.ConditionerParameters.toObject(includeInstance, f)
conditioner: (f = msg.getConditioner()) && proto.gooseai.ConditionerParameters.toObject(includeInstance, f),
dryRun: jspb.Message.getBooleanFieldWithDefault(msg, 9, false)
};

if (includeInstance) {
Expand Down Expand Up @@ -6971,6 +7020,10 @@ proto.gooseai.Request.deserializeBinaryFromReader = function(msg, reader) {
reader.readMessage(value,proto.gooseai.ConditionerParameters.deserializeBinaryFromReader);
msg.setConditioner(value);
break;
case 9:
var value = /** @type {boolean} */ (reader.readBool());
msg.setDryRun(value);
break;
default:
reader.skipField();
break;
Expand Down Expand Up @@ -7061,6 +7114,13 @@ proto.gooseai.Request.serializeBinaryToWriter = function(message, writer) {
proto.gooseai.ConditionerParameters.serializeBinaryToWriter
);
}
f = message.getDryRun();
if (f) {
writer.writeBool(
9,
f
);
}
};


Expand Down Expand Up @@ -7304,6 +7364,24 @@ proto.gooseai.Request.prototype.hasConditioner = function() {
};


/**
* optional bool dry_run = 9;
* @return {boolean}
*/
proto.gooseai.Request.prototype.getDryRun = function() {
return /** @type {boolean} */ (jspb.Message.getBooleanFieldWithDefault(this, 9, false));
};


/**
* @param {boolean} value
* @return {!proto.gooseai.Request} returns this
*/
proto.gooseai.Request.prototype.setDryRun = function(value) {
return jspb.Message.setProto3BooleanField(this, 9, value);
};



/**
* List of repeated fields within this message type.
Expand Down
68 changes: 34 additions & 34 deletions gooseai/generation/generation_pb2.py
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion gooseai/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/stability-ai/api-interfaces/gooseai
go 1.18

require (
google.golang.org/grpc v1.49.0
google.golang.org/grpc v1.50.1
google.golang.org/protobuf v1.28.1
)

Expand Down
4 changes: 2 additions & 2 deletions gooseai/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEY
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.49.0 h1:WTLtQzmQori5FUH25Pq4WT22oCsv8USpQ+F6rqtsmxw=
google.golang.org/grpc v1.49.0/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI=
google.golang.org/grpc v1.50.1 h1:DS/BukOZWp8s6p4Dt/tOaJaTQyPyOoCcrjroHuCeLzY=
google.golang.org/grpc v1.50.1/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
Expand Down
2 changes: 1 addition & 1 deletion gooseai/project/project.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/proto/generation.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,7 @@ message Answer {
uint64 created = 4;
optional AnswerMeta meta = 6;
repeated Artifact artifacts = 7;
optional double estimated_cost = 8; // Only set if Request.dry_run is true.
Copy link
Contributor

@arsenetar arsenetar Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm partial to just calling this cost which would allow it to be used both for dry_run estimates as well as actual cost if desired. dry_run already implies estimation / non-actual results.

Copy link
Contributor

@arsenetar arsenetar Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Units for this need to be documented somewhere as well ($ or credits).

Copy link
Contributor Author

@johnsabath johnsabath Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm partial to just calling this cost which would allow it to be used both for dry_run estimates as well as actual cost if desired. dry_run already implies estimation / non-actual results.

I like that.

Units for this need to be documented somewhere as well ($ or credits).

Definitely. I'm waiting for feedback on which units we want to use here, so it's deliberately vague at the moment.

Copy link
Contributor

@wbrown wbrown Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want credits here. And we should be able to return the cost of a Request in an Answer regardless of whether or not dry run is set.

Copy link
Contributor Author

@johnsabath johnsabath Nov 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We definitely could.

Do y'all feel that in a multi-answer scenario, returning the cost on each Answer could be confusing?

Copy link
Contributor

@arsenetar arsenetar Nov 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think returning the cost with the respective answer would be the appropriate way of doing it and less confusing than returning the cost elsewhere.

Copy link
Contributor Author

@johnsabath johnsabath Nov 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where I see the confusion coming from is that we're computing the cost for a single request, but returning multiple answers.

Do the answers report their share of the cost, or do they report the total cost of the request?

Copy link
Contributor

@arsenetar arsenetar Nov 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose regardless of the exact desired behavior the cost needs to be within an answers object since that is the message used for the responses. Either each answer could have partial cost, or the total cost could be reported by the last answer sent back to the client. I think either one could be appropriate. Regardless of that implementation detail, I think we want to move forward with labeling this cost and per other discussions using credits as the unit of measure here.

}

// A Request is a set of Artifacts, which can be of any type with model or
Expand All @@ -302,6 +303,9 @@ message Request {
AssetParameters asset = 8;
}
optional ConditionerParameters conditioner = 6;
// If set to true, generation will not be performed, instead the request will be
// validated and a single Answer with an estimated credit cost will be returned.
bool dry_run = 9;
}

//
Expand Down