cloud phone POC framework: how to evaluate in 2 weeks
cloud phone POC framework: how to evaluate in 2 weeks
a cloud phone POC framework done in two weeks beats a six-week one almost every time. the longer the proof of concept drags, the more political it gets, the more your evaluator forgets the criteria, and the more the vendor’s sales engineer subtly reshapes the test to match their strengths. this guide gives you a strict 14-day plan with day-by-day tasks, named owners, and a five-criteria scorecard so you exit the POC with a clear winner.
if you are still in vendor-shortlist mode, the vendor checklist and the RFP template come first. this article picks up after the shortlist of two finalists.
why two weeks and not four
three reasons.
- attention decay. by week four, evaluators have moved to other priorities and the POC stalls.
- vendor fatigue. their sales engineer eventually disengages too. you will not get answers as fast.
- decision quality. the meaningful failures show up in the first week if you front-load the test cases. extending the timeline rarely surfaces new signal.
teams that need longer should split into POC + paid pilot. POC = 2 weeks, contractual go/no-go. pilot = 30-60 days at a discounted rate to validate scale and integration depth.
the team
four roles, minimum.
- POC lead (1 person, 50% time). owns the timeline, scorecard, daily standup with the vendor.
- technical evaluator (1-2 engineers, 25% time). runs test cases, files defects, checks integration paths.
- security reviewer (1 person, 10% time). validates compliance answers, reviews the audit log, checks data handling.
- finance/procurement liaison (1 person, 10% time). builds the TCO comparison from real usage data captured during the POC.
without the POC lead, the framework collapses. that role is non-negotiable.
the 14-day plan
days 1-2: setup and baseline
- vendor provisions a tenant with admin access for POC lead and technical evaluator
- POC lead distributes the scorecard, kicks off a shared Slack/Teams channel with vendor SE
- technical evaluator runs the smoke-test checklist: lock device, install APK, screenshot, unlock, repeat with three different device models
deliverable end of day 2: a one-page “this is real” memo confirming the platform exists and basic flow works.
days 3-5: integration work
- wire the platform into your CI (see the GitLab CI guide or the GitHub Actions equivalent)
- run your real test suite for one shippable feature against the cloud phones
- record wall-clock time, flake rate, total minutes consumed, and total dollar cost
deliverable end of day 5: a CI run log showing integration works end-to-end on real product code, not vendor demo apps.
days 6-8: scale and stress
- run a parallel test sharded across 4-8 devices simultaneously
- intentionally cancel jobs mid-flight and verify devices unlock cleanly
- simulate a flaky network from one of your CI runners and observe ADB resilience
- pull 24 hours of audit logs and verify all admin actions are present
deliverable end of day 8: a “scale and resilience” report covering parallel throughput, recovery behavior, and audit completeness.
days 9-10: security and compliance
- security reviewer walks the SOC 2 / ISO report, confirms regions match your data residency
- run a wipe-between-sessions test: install an APK on session 1, lock as a different user in session 2, verify the APK is gone
- enumerate every IAM role, confirm you can build a least-privilege custom role
- run a SAML or OIDC SSO test against your IdP
deliverable end of day 10: a security sign-off memo (or list of blockers).
days 11-12: cost reconciliation
- finance liaison takes actual usage data from the POC and runs the TCO worksheet for both vendors at projected production scale (10x, 50x, 100x)
- vendor SE provides quote for committed pricing tier and annual prepay discount
deliverable end of day 12: a TCO comparison sheet covering Y1, Y2, Y3 at three scale points.
days 13-14: decision and exit
- POC lead aggregates scorecards from technical, security, finance
- vendor SE answers final clarification questions in writing
- decision meeting: 60 minutes max, all evaluators present, one of three outcomes (go, no-go, paid pilot)
- if no-go, fire the off-boarding letter same day so vendor stops investing time
deliverable end of day 14: signed decision memo plus contract draft (if go) or thank-you-and-goodbye letter (if no-go).
the five-criteria scorecard
each evaluator scores 1-5 across five categories. weighted sums determine the winner.
| criterion | weight | what to look for |
|---|---|---|
| basic flow reliability | 20% | lock, ADB, unlock works 100% of the time across 50+ trials |
| integration ease | 20% | wiring into CI takes <1 day with their docs |
| scale behavior | 20% | 8-way parallel runs cleanly, no device contention errors |
| security posture | 20% | SOC 2 valid, audit log complete, wipe verified |
| TCO at scale | 20% | three-year cost predictable, no surprise line items |
minimum passing score per criterion: 3. any 1 or 2 in any category is a hard stop until clarified.
the test cases that actually predict production
not all tests carry equal signal. these five separate winners from losers.
- the cancelled-job test. kill a CI job mid-run 10 times. count how many devices end up stuck in a locked state. zero is a pass; one or more is a tell.
- the cross-region test. lock devices from a CI runner in a different region than the device. measure flake rate. if it doubles vs co-located, the platform’s network handling is brittle.
- the wipe-between-sessions test. verify session 2 truly does not see session 1’s installed APKs, accounts, or files. a fail here kills enterprise deals.
- the audit-log completeness test. make 20 admin actions, then export the audit log. verify all 20 appear with full context. missing entries are a security red flag.
- the bill-prediction test. run a planned workload, predict the bill, then check the actual invoice. variance >10% means the pricing model has surprises you do not yet understand.
what to do if both vendors pass
happens 30% of the time. tiebreakers, in order:
- which vendor’s audit log was more complete?
- which vendor’s support response time was faster during the POC?
- which vendor’s contract had cleaner exit terms?
- which vendor’s three reference customers gave the most concrete answers?
if still tied, pick the cheaper one and extend a six-month try clause in the MSA so you can switch with minimal pain.
frequently asked questions
what if the vendor wants to extend the POC by another two weeks?
push back. either there are clear blockers (in which case file them as defects with severity, expected resolution date), or the vendor is fishing for more time to influence the decision. extensions usually favor the vendor, not you.
should the vendor be in the daily standup?
yes for days 1-5 and 11-12. no for the security review and decision meeting. you need uncoached space to talk among yourselves.
can I run a POC against three vendors at once?
possible but exhausting. two is the sweet spot. three works only if you have a dedicated POC team with no other priorities.
what if I find a critical security gap mid-POC?
stop the POC, write up the gap with evidence, share with the vendor, ask for a written remediation plan with date. resume only if the plan is credible. if not, no-go.
should sales discounts be on the table during the POC?
ask once at day 11-12 when finance is doing the TCO. do not let sales hijack the technical and security tracks before then.
ready to start a real two-week POC instead of a four-month one? open a cloudf.one trial on day 1 and you have your first vendor in flight by lunch.