agentsec-eval

v0.55.0 suspicious
4.0
Medium Risk

Security assessment framework for AI agents — adversarial test runner + server-side audit + scoring

🤖 AI Analysis

Final verdict: SUSPICIOUS

The package is flagged as suspicious due to its metadata indicating incomplete maintainer information and suspicious links, despite showing no signs of obfuscation or credential harvesting.

  • Metadata risk identified with incomplete maintainer information and suspicious links
  • No evidence of obfuscation or credential harvesting
Per-check LLM notes
  • Obfuscation: No obfuscation patterns detected, indicating low risk.
  • Credentials: No credential harvesting patterns detected, indicating low risk.
  • Metadata: Suspicious links and incomplete maintainer information suggest potential risk.

📦 Package Quality Overall: Low (4.6/10)

◈ Medium Test Suite 6.0

Partial test coverage signals detected

  • Test runner config found: pyproject.toml
◈ Medium Documentation 5.0

Some documentation present

  • Detailed PyPI description (42029 chars)
○ Low Contributing Guide 4.0

No contributing guide or governance files found

  • Development Status classifier >= Beta
◈ Medium Type Annotations 7.0

Partial type annotation coverage

  • Type checker (mypy / pyright / pytype) referenced in project
  • 155 type-annotated function signatures detected in source
○ Low Multiple Contributors 1.0

Could not retrieve contributor data from GitHub

  • GitHub API error: 404

🔬 Heuristic Checks

Outbound Network Calls score 4.5

Found 3 network call pattern(s)

  • ient_factory() return httpx.AsyncClient(timeout=self._timeout) async def send(self, message: st
  • qa: S101 client = paramiko.SSHClient() client.load_system_host_keys() cli
  • amiko.SSHClient: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_
Code Obfuscation

No obfuscation patterns detected

Shell / Subprocess Execution score 2.0

Found 1 shell execution pattern(s)

  • =sys.stderr) result = subprocess.run(cmd, cwd=REPO_ROOT, check=False) # noqa: S603 if re
Credential Harvesting

No credential harvesting patterns detected

Typosquatting

No typosquatting candidates detected

Registered Email Domain

Email domain looks legitimate: gmail.com>

Suspicious Page Links score 4.0

Found 2 suspicious link(s) on the package page

  • Non-HTTPS external link: http://your-agent-host:8080
  • Non-HTTPS external link: http://127.0.0.1:8080/v1
Git Repository History score 3.0

Repository not found (deleted or private)

  • Repository not found (deleted or private)
Maintainer History score 4.0

2 maintainer concern(s) found

  • Author name is missing or very short
  • Author "" appears to have only 1 package on PyPI (new or inactive account)
Known CVE Vulnerabilities

No known vulnerabilities found in OSV database.

💡 AI App Starter Prompt

Use this prompt to build a project with agentsec-eval
Create a security assessment tool named 'AgentSecGuard' using the Python package 'agentsec-eval'. This tool will serve as a comprehensive platform for evaluating the security resilience of various AI agents against common adversarial attacks. The application should include the following core functionalities:

1. **Adversarial Test Runner**: Implement a feature that allows users to define and run custom adversarial tests on AI agents. These tests could simulate different types of attacks such as data poisoning, model evasion, or inference attacks.
2. **Server-Side Audit**: Develop an auditing system that logs and analyzes the performance of AI agents under attack conditions. This system should track metrics like attack success rate, time taken to recover from an attack, and any changes in model accuracy post-attack.
3. **Scoring Mechanism**: Integrate a scoring system that evaluates the overall security posture of AI agents based on their performance during adversarial tests. The score should consider factors such as robustness against attacks, recovery time, and effectiveness of defense mechanisms.
4. **User Interface**: Design a simple yet intuitive user interface where users can input details about the AI agents they want to test, select or customize adversarial scenarios, and view results in real-time.
5. **Report Generation**: Include functionality to generate detailed reports summarizing the findings from each security assessment. Reports should be customizable and capable of exporting data in formats like PDF or CSV.

To utilize the 'agentsec-eval' package, follow these steps:
- Install the package using pip.
- Import necessary modules from the package to handle adversarial testing, auditing, and scoring.
- Use the provided APIs to integrate these functionalities into your application.
- Customize the package’s components to fit the specific needs of 'AgentSecGuard', such as adapting the scoring mechanism to prioritize certain aspects of security over others.

Ensure that the final product is well-documented, with clear instructions on how to set up and use the tool, as well as how to contribute to its development.