Devin AI Capabilities and Limitations

Pros & Cons of Using AI for Strategic What-If Scenario Planning While AI assistants like ChatGPT are still largely unexplored for marketing strategy development, forward-thinking marketers should start using their capabilities in this space. Recently, I used ChatGPT to help with strategic planning and provided it with (redacted and generalized) data from the previous year, including budget allocation across tactics for different segments (SMB, mid-market, and enterprise), performance goals, as well as campaign and incremental revenue results. ChatGPT offered solid qualitative responses to my "what-if" scenario inquiries, such as: ► What would be the impact of shifting 10% of our advertising budget to ABM (Account-Based Marketing) in the enterprise segment? ► How might we adjust our marketing strategy across all tactics to maintain ROI during a market downturn affecting SMBs? ► With a flat budget and a 20% cut in advertising spend (reallocated to digital and partnerships), what are the potential impacts and considerations? ► What's the recommended marketing mix for the following year to meet ROI and hurdle rate goals in each segment? However, ChatGPT struggled with providing precise numerical outputs for reallocated budget mixes that met financial goals within a flat spend. Despite multiple attempts, the numbers didn't add up, reflecting the current limitation of language models in handling complex mathematical computations. See conversation screenshots in the carousel below. While AI assistants excel at pattern recognition, understanding mathematical concepts remains a challenge. They may know that 2 + 2 = 4 from exposure to that pattern, but lack the knowledge of addition as a concept. This doesn't mean you can't use AI for strategic scenario planning. In fact, these tools can be great thought partners for qualitative "what-if" analyses. Just don't expect precise mathematical outputs, especially complex ones, from them yet. As AI capabilities rapidly evolve, these limitations will likely improve. But for now, it's important to understand their strengths and weaknesses for this type of use case. Have you used AI for strategic scenario planning? What has your experience been? Feel free to DM me or Tahnee Perry if you want to jumpstart AI adoption and inspire your teams with what's possible through applied AI use cases for content creation and beyond (e.g., ideation & collaboration, automation, analytics & research, and personalization). Here's a small sampling of use cases for your reference, https://lnkd.in/gw5Vpf6b #StrategicMarketing #ScenarioPlanning #AICollaboration #AIAnalytics #AIUseCases GrowthPath Partners

Crafting AI Agents: Real-World Lessons from the Front Lines As many of us have been diving into the world of AI agents, it’s increasingly clear that the path to creating effective agents may not be so simple. It’s a far cry from the launch of ChatGPT when so many of us thought that AI would soon be able to do everything and we were fast approaching AGI.  At Bito, our foray into developing our Code Review Agent and other agent work has been a real journey of discovery, highlighting key considerations for anyone looking to integrate AI into their operations. This isn't just about pioneering technology; it's about building tools that work reliably under the stringent demands of enterprises. Let's talk about some of the issues we've seen:    The Quest for Consistency Consistency is the cornerstone of trust in any AI implementation. Imagine an AI that reviews code — one day it flags a security issue, and the next, it overlooks the same flaw. This kind of inconsistency is a deal-breaker in code review, where reliability isn't just preferred; it's imperative.   Hallucinations – Imagination Run Amok In an enterprise context, business users are not excited to see AI outputs that are completely made up or based on entirely wrong assumptions that the AI made. For example, in the developer context, asking the AI what a function does and getting a completely fictional response that is completely incorrect is simply not acceptable. It erodes trust in any output the tool is providing (now you don’t know which outputs you can trust and which you can’t).    Tackling Complex Reasoning The ability of an AI to navigate complex reasoning tasks is both its greatest strength and its most notable weakness. In creating AI agents, we've encountered scenarios where the AI excels in certain steps only to falter at others, revealing a gap in its ability to maintain a coherent chain of reasoning throughout a task. This limitation becomes particularly evident in structured output generation (such as a JSON), where a single flaw can derail the entire output for some other agent to handle. Addressing this can be difficult and leads us to our next big learning.     A Hybrid Solution The evolution of our approach at Bito has led us to a hybrid model, merging traditional coding techniques with AI's dynamic capabilities. Initially, the industry buzz suggested leaning heavily on AI, expecting it to carry the brunt of the workload. However, reality taught us differently. By anchoring our agents in code—allocating 70% to 80% of the task to traditional programming, supplemented by AI for specific reasoning tasks—we've achieved a significant leap in both reliability and output quality. Read my whole post here, including how we combat many of the issues above: https://lnkd.in/gnKD_hpU How do these learnings we’ve had at Bito compare with your agent experiences? Please do share in the comments. 

From InsideBigData (12/12/2023) AI is Still Too Limited to Replace People. Commentary by Mica Endsley, a Fellow of the Human Factors and Ergonomics Society (HFES) “NVIDA’s CEO Jensen Huang declared that AI will be “fairly competitive” with people within five years, echoing the rolling “it’s just around the corner” claim we have been hearing for decades. But this view neglects the very real challenges AI is up against. AI has made impressive gains due to improvements in machine learning as well as access to large data sets. Extending these gains to many real-world applications in the natural world remains challenging, however. Tesla and Cruise’s automated vehicle accidents point to the difficulties of implementing AI in high-consequence domains such as military, aviation, healthcare, and power operations. Most importantly, AI struggles to deal with novel situations that it is not trained on. The National Academy of Sciences recently released a report on “Human-AI Teaming” documenting AI technical limitations that stem from brittleness, perceptual limitations, hidden biases, and lack of a model of causation that is crucial for understanding and predicting future events. To be successful, AI systems must become more human-centered. AI rarely fully replaces people; Instead, it must successfully interact with people to provide its potential benefits. But when the AI is not perfect, people struggle to compensate for its shortcomings. They tend to lose situation awareness, their decisions can become biased by inaccurate AI recommendations, and they struggle with knowing when to trust it and when not to. The Human Factors and Ergonomics Society (HFES) developed a set of AI guardrails to make it safe and effective, including the need for AI to be both explainable and transparent in real-time regarding its ability to handle current and upcoming situations and predictability of its actions. For example, ChatGPT provides excellent language capabilities but very low transparency regarding the accuracy of its statements. Misinformation is mixed in with accurate information with no clues as to which is which. Most AI systems still fail to provide users with the insights they need; a problem that is compounded when capabilities change over time with learning. While it may be some time before AI can truly act alone, it can become a highly useful tool when developed to support human interaction.” https://lnkd.in/gvVN2XD4

In recent months, I have had the pleasure of contributing to the International Scientific Report on the Safety of Advanced AI, a project of the UK government's Department for Science, Innovation and Technology (DSIT) and AI Safety Institute. This report sets out an up-to-date, science-based understanding of the safety of advanced AI systems. The independent, international, and inclusive report is a landmark moment of international collaboration. It marks the first time the international community has come together to supports efforts to build a shared scientific and evidence-based understanding of frontier AI risks. The intention to create such a report was announced at the AI Safety Summit in November 2023 This interim report is published ahead of the AI Seoul Summit next week. The final report will publish before the AI Action Summit in France. The interim report restricts its focus to a summary of the evidence on general-purpose AI, which have advanced rapidly in recent years. The report synthesizes the evidence base on the capabilities of, and risks from, general-purpose AI and evaluates technical methods for assessing and mitigating them. Key report takeaways include: 1️⃣ General-purpose AI can be used to advance the public interest, leading to enhanced wellbeing, prosperity, and scientific discoveries. 2️⃣ According to many metrics, the capabilities of general-purpose AI are advancing rapidly. Whether there has been significant progress on fundamental challenges such as causal reasoning is debated among researchers. 3️⃣ Experts disagree on the expected pace of future progress of general-purpose AI capabilities, variously supporting the possibility of slow, rapid, or extremely rapid progress. 4️⃣ There is limited understanding of the capabilities and inner workings of general-purpose AI systems. Improving our understanding should be a priority. 5️⃣ Like all powerful technologies, current and future general-purpose AI can be used to cause harm. For example, malicious actors can use AI for large-scale disinformation and influence operations, fraud, and scams. 6️⃣ Malfunctioning general-purpose AI can also cause harm, for instance through biassed decisions with respect to protected characteristics like race, gender, culture, age, and disability. 7️⃣ Future advances in general-purpose AI could pose systemic risks, including labour market disruption, and economic power inequalities. Experts have different views on the risk of humanity losing control over AI in a way that could result in catastrophic outcomes. 8️⃣ Several technical methods (including benchmarking, red-teaming and auditing training data) can help to mitigate risks, though all current methods have limitations, and improvements are required. 9️⃣ The future of AI is uncertain, with a wide range of scenarios appearing possible. The decisions of societies and governments will significantly impact its future. #ResponsibleAI #GenerativeAI #ArtificialIntelligence #AI #AISafety