Class: Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1SingleReinforcementTuningRewardConfig

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
lib/google/apis/aiplatform_v1beta1/classes.rb,
lib/google/apis/aiplatform_v1beta1/representations.rb,
lib/google/apis/aiplatform_v1beta1/representations.rb

Overview

SingleReinforcementTuningRewardConfig defines a single reward function configuration for RL tuning. Each reward calculation/evaluation consists of two stages: 1. Stage 1: Parses the part of information important from sample response via regex extract, or simply takes the sample response unmodified. 2. Stage 2: Calls the configured reward scorer to compute the reward.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(**args) ⇒ GoogleCloudAiplatformV1beta1SingleReinforcementTuningRewardConfig

Returns a new instance of GoogleCloudAiplatformV1beta1SingleReinforcementTuningRewardConfig.



59310
59311
59312
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59310

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#autorater_scorer ⇒ Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1ReinforcementTuningAutoraterScorer

ReinforcementTuningAutoraterScorer is used to score parsed responses for classification based autorater use cases. For example, for math problems, users can use classification based autorater to calculate rewards based on the autorater parsed response against a reference answer. Corresponds to the JSON property autoraterScorer

Returns:



59230
59231
59232
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59230

def autorater_scorer
  @autorater_scorer
end

#cloud_run_reward_scorer ⇒ Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1ReinforcementTuningCloudRunRewardScorer

ReinforcementTuningCloudRunRewardScorer allows users to implement a reward function through GCP Cloud Run. Comparing with ReinforcementTuningCodeExecutionRewardScorer that runs in a Sandbox and has no internet access, Cloud Run reward scorer is fully controlled by users. The Cloud Run service should implement the following HTTP API: HTTP method: POST HTTP request body: ` "example": ReinforcementTuningExample, "response": Content, "metadata": ` "step": int "tuning_job_id": int64 ` ` * example is a ReinforcementTuningExample in ProtoJSON format, (i.e., the format is the same as as one line in the training/validation dataset except that the keys must be in camel case). System instructions (i.e., example.get(" systemInstruction")) and references (i.e., example.get("references")) are also included in the example provided that they are set in the training/ validation dataset. * response is a Content in ProtoJSON format (i.e., keys must be in camel case), which is the same as the Online Prediction response for Gemini models. HTTP response body: "reward": float, " user_requested_aux_info": str // Optional where the field " user_requested_aux_info" is any (optional) string provided by users for assisting debugging. It's in snake case. This field is mostly useful when calling the GenAiTuningService.ValidateReinforcementTuningReward API, where the proto field (not Cloud Run HTTP response body) userRequestedAuxInfo will be populated if the Cloud Run reward function sets this field in the HTTP response. The following are examples for the HTTP request and response body. Example HTTP request body: ` "example": ` "contents": [ ` "role": "user", " parts": [ ` "text": "What is the capital of France?" ` ] ` ], "references": ` " answer": "Paris" ` `, "response": ` "parts": [ ` "text": "London" ` ] `, " metadata": ` "step": 1, "tuning_job_id": 123456789 ` ` Example HTTP response body: ` "reward": -1.0 ` Note: Reward output by Cloud Run reward function is clipped to be within [-1, 1], i.e., reward = max(min( reward, 1.0), -1.0). Corresponds to the JSON property cloudRunRewardScorer

Returns:



59263
59264
59265
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59263

def cloud_run_reward_scorer
  @cloud_run_reward_scorer
end

#code_execution_reward_scorer ⇒ Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1ReinforcementTuningCodeExecutionRewardScorer

ReinforcementTuningCodeExecutionRewardScorer allows users to implement a function to evaluate rewards for the sample response. The function signature is as follows: def evaluate(example: dict[str, Any], response: dict[str, Any]) -> float: ... example is a ReinforcementTuningExample in ProtoJSON format, (i.e., the format is the same as as one line in the training/ validation dataset except that the keys must be in camel case). System instructions (i.e., example.get("systemInstruction")) and references (i.e., example.get("references")) are also included in the example provided that they are set in the training/validation dataset. response is a Content in ProtoJSON format (i.e., keys must be in camel case), which is the same as the Online Prediction response for Gemini models. Note: Reward output by the evaluate function is clipped to be within [-1, 1], i.e., reward = max(min( reward, 1.0), -1.0). Corresponds to the JSON property codeExecutionRewardScorer

Returns:



59280
59281
59282
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59280

def code_execution_reward_scorer
  @code_execution_reward_scorer
end

#parse_response_config ⇒ Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1ReinforcementTuningParseResponseConfig

Defines how to parse sample response config for reinforcement tuning. The parsed response (i.e., substring) will be passed to the reward functions. For example, the input prompt might be: > "Perform step-by-step thoughts first to problem A, finally output answer in the block." The sample response from the model under tuning might look like: > "Yes" Here, users can define the following parse config: ` "parseType": "REGEX_EXTRACT" , "regexExtractExpression": ".*(.*?)" ` The resulting parsed response would be "Yes" and will be passed to the reward functions for evaluating rewards. Corresponds to the JSON property parseResponseConfig

Returns:



59293
59294
59295
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59293

def parse_response_config
  @parse_response_config
end

#reward_name ⇒ String

A unique reward name for identifying each single reinforcement tuning reward. Corresponds to the JSON property rewardName

Returns:

  • (String) 


59298
59299
59300
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59298

def reward_name
  @reward_name
end

#string_match_reward_scorer ⇒ Google::Apis::AiplatformV1beta1::GoogleCloudAiplatformV1beta1ReinforcementTuningStringMatchRewardScorer

ReinforcementTuningStringMatchRewardScorer is used to score parsed responses for string matching use cases. For example, for math problems, users can use string match scorer to check if the correct exact answer is generated. Note: Reward returned by the string match reward function is clipped to be within [- 1, 1] if wrongAnswerReward or correctAnswerReward are beyond the range, i.e., reward = max(min(reward, 1.0), -1.0). Corresponds to the JSON property stringMatchRewardScorer

Returns:



59308
59309
59310
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59308

def string_match_reward_scorer
  @string_match_reward_scorer
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object



59315
59316
59317
59318
59319
59320
59321
59322
 
# File 'lib/google/apis/aiplatform_v1beta1/classes.rb', line 59315

def update!(**args)
  @autorater_scorer = args[:autorater_scorer] if args.key?(:autorater_scorer)
  @cloud_run_reward_scorer = args[:cloud_run_reward_scorer] if args.key?(:cloud_run_reward_scorer)
  @code_execution_reward_scorer = args[:code_execution_reward_scorer] if args.key?(:code_execution_reward_scorer)
  @parse_response_config = args[:parse_response_config] if args.key?(:parse_response_config)
  @reward_name = args[:reward_name] if args.key?(:reward_name)
  @string_match_reward_scorer = args[:string_match_reward_scorer] if args.key?(:string_match_reward_scorer)
end