awb

v1.4.0 suspicious
4.0
Medium Risk

Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.

🤖 AI Analysis

Final verdict: SUSPICIOUS

The package is flagged as potentially suspicious due to shell execution risks and a possible typosquatting attempt targeting 'arq'. However, it has low risks in other areas such as network calls, obfuscation, and credential handling.

  • Shell execution attempts
  • Possible typosquatting
Per-check LLM notes
  • Network: No network calls were detected.
  • Shell: Shell execution attempts to check the version of external tools ('aider', 'claude') might indicate dependency checks but could also signify potential execution of untrusted commands.
  • Obfuscation: No obfuscation patterns detected, indicating low risk of malicious intent.
  • Credentials: No credential harvesting patterns detected, indicating low risk of secret theft.
  • Metadata: The maintainer has only one package, which could indicate a new or less active account.
  • ⚠ Typosquatting target: arq

📦 Package Quality Overall: Medium (5.6/10)

◈ Medium Test Suite 6.0

Partial test coverage signals detected

  • Test runner config found: pyproject.toml
◈ Medium Documentation 7.0

Some documentation present

  • Documentation URL: "Documentation" -> https://github.com/xmpuspus/ai-workflow-benchmark/blob/main/
  • Detailed PyPI description (28388 chars)
○ Low Contributing Guide 2.0

No contributing guide or governance files found

  • No CONTRIBUTING, CODE_OF_CONDUCT, or governance files found
◈ Medium Type Annotations 5.0

Partial type annotation coverage

  • 186 type-annotated function signatures detected in source
✦ High Multiple Contributors 8.0

Active multi-contributor project

  • 3 unique contributor(s) across 88 commits in xmpuspus/ai-workflow-benchmark
  • Small but multi-author team (3–4 contributors)

🔬 Heuristic Checks

✓ Outbound Network Calls

No suspicious network call patterns found

✓ Code Obfuscation

No obfuscation patterns detected

⚠ Shell / Subprocess Execution score 10.0

Found 6 shell execution pattern(s)

  • try: r = subprocess.run(["aider", "--version"], capture_output=True, text=True, time
  • lf) -> bool: result = subprocess.run(["which", "claude"], capture_output=True, timeout=10)
  • try: result = subprocess.run( ["claude", "--version"], capture_output=Tru
  • et_env() result = subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=30)
  • try: result = subprocess.run( ["codex", "--version"], capture_output=True
  • try: result = subprocess.run( ["gh", "extension", "list"], capture_output
✓ Credential Harvesting

No credential harvesting patterns detected

⚠ Typosquatting score 3.0

Possible typosquat of: arq

  • "awb" is 2 edit(s) from "arq"
✓ Registered Email Domain

No author email provided

✓ Suspicious Page Links

All external links appear legitimate

✓ Git Repository History

Repository xmpuspus/ai-workflow-benchmark appears legitimate

⚠ Maintainer History score 2.0

1 maintainer concern(s) found

  • Author "Xavier Puspus" appears to have only 1 package on PyPI (new or inactive account)
✓ Known CVE Vulnerabilities

No known vulnerabilities found in OSV database.

💡 AI App Starter Prompt

Use this prompt to build a project with awb 
Develop a comprehensive benchmarking tool for AI coding assistants using the 'awb' Python package. Your goal is to create a utility that evaluates different AI coding tools based on their performance across various coding tasks, rather than just assessing their inherent capabilities. This tool will be particularly useful for developers looking to integrate AI into their workflows, as it will provide insights into how effectively these tools can assist in real-world scenarios.

### Key Features:
- **Task Library**: Integrate a library of 100 predefined coding tasks covering a wide range of programming challenges and complexities.
- **Sigmoid Scoring System**: Implement a scoring system that uses a sigmoid function to evaluate the performance of each AI tool on each task, providing a normalized score between 0 and 1.
- **Performance Metrics**: Measure AI tools across 12 different capability dimensions such as code generation speed, accuracy, maintainability, etc.
- **Gap Analysis**: Provide a comparative analysis of the strengths and weaknesses of each AI tool, highlighting areas where they excel and areas needing improvement.
- **User Interface**: Design a simple yet effective command-line interface for users to input the AI tools they want to test and view the results.
- **Customization Options**: Allow users to customize the benchmarking process by selecting specific tasks or metrics to focus on.

### Utilizing the 'awb' Package:
- Use the 'awb' package to set up the benchmarking framework, ensuring that all tasks and metrics are correctly defined according to the package's standards.
- Leverage the package's built-in functions to automate the execution of the tasks and the collection of performance data.
- Apply the sigmoid scoring mechanism provided by 'awb' to ensure fair and consistent evaluation of each AI tool.
- Employ 'awb' for generating detailed reports and visualizations that summarize the performance of each AI tool, aiding in the interpretation of the benchmarking results.

💬 Discussion Feed

Leave a comment

No discussion yet. Be the first to share your thoughts!