Getting Started with AutoGenAI

AutoGenAI treats a generative AI workflow as a configuration containing a prompt, a provider, and a generation strategy. The package can benchmark these configurations using a task-specific scorer and then select configurations under quality, cost, and latency objectives.

Offline example

library(AutoGenAI)
ex <- autogenai_example()
fit <- optimize_ai(
  ex$task,
  ex$data,
  ex$providers,
  ex$prompts,
  temperatures = 0,
  strategies = "single"
)
fit
#> AutoGenAI optimization
#> Task: sentiment 
#> Configurations: 12 
#> 
#> Best configuration
#>  Provider: mock-balanced 
#>  Prompt ID: 1 
#>  Strategy: single 
#>  Temperature: 0 
#>  Quality: 1.0000 
#>  Cost: 0.0000335 
#>  Latency: 0.0000 sec
#>  Utility: 0.9397

Pareto-efficient choices

pareto_ai(fit)
#>                        config_id prompt_id
#> 1 1::mock-balanced::0::single::1         1
#> 9    1::mock-cheap::0::single::1         1
#>                                                                         prompt
#> 1 Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#> 9 Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#>        provider temperature strategy samples   quality structured_score
#> 1 mock-balanced           0   single       1 1.0000000               NA
#> 9    mock-cheap           0   single       1 0.6666667               NA
#>   success_rate     cost latency n   utility robustness
#> 1            1 3.35e-05       0 6 0.9396759         NA
#> 9            1 6.70e-06       0 6 0.3125000         NA

Robustness

st <- stress_test(
  ex$task,
  ex$data,
  ex$providers[[1]],
  ex$prompts[[1]]
)
st
#>   perturbation quality
#> 1     original       1
#> 2   whitespace       1
#> 3         case       1
#> 4         typo       1
#> 5   distractor       1
robustness_score(st)
#> [1] 1

Real providers

A real provider is any R function accepting prompt, input, and params and returning one text value. This deliberately keeps model-specific credentials and network behavior outside the package core.