Enterprise Scale Testing for Partners

/s/image03.jpg?v=1Designing for high performance is a critical part of the success of ISV applications. Don’t let your largest and most important customers find performance issues with your app before you do. As you design and build your app, ask yourselves these questions: 

  • Does my application perform well for a large number of concurrent users? 
  • Can my application scale to elegantly handle a large amount of data?
Large Data Volume testing is an aspect of this performance testing for a large amount of data (millions of records) in an org with a significant load (thousands) of concurrent users. 

Value for ISVs
  • Build Enterprise class applications 
  • Confidence that your apps will Scale 
  • Competitive advantage in marketplace 
  • Better performance = Increased Revenue
As a Salesforce partner, please review the demo videos (see "Media"), the What You Need to Know to understand the steps for performance testing, Resources and an FAQ. 

Additional questions? Engage with us in the Enterprise Scale Collaboration Group. 
  • - All partner contracts state that any public mention of salesforce.com or any of its brands MUST be approved by Salesforce
  • - All materials must be submitted via our Google form to obtain approval for any public mention of Salesforce.
Submit content via Google Form >
    • - Approval of partner press releases can take up to 14 days for Summit and Crest level partners, and up to 30 days for all other partners. 
  • - All partner contracts state that any public mention of salesforce.com or any of its brands MUST be approved by Salesforce
  • - All materials must be submitted via our Google form to obtain approval for any public mention of Salesforce.
Submit content via Google Form >
    • - Approval of partner press releases can take up to 14 days for Summit and Crest level partners, and up to 30 days for all other partners. 
  • - All partner contracts state that any public mention of salesforce.com or any of its brands MUST be approved by Salesforce
  • - All materials must be submitted via our Google form to obtain approval for any public mention of Salesforce.
Submit content via Google Form >
    • - Approval of partner press releases can take up to 14 days for Summit and Crest level partners, and up to 30 days for all other partners. 
  • - All partner contracts state that any public mention of salesforce.com or any of its brands MUST be approved by Salesforce
  • - All materials must be submitted via our Google form to obtain approval for any public mention of Salesforce.
Submit content via Google Form >
    • - Approval of partner press releases can take up to 14 days for Summit and Crest level partners, and up to 30 days for all other partners. 
Scale Testing
Scale Testing is testing applications to validate if they perform well in Salesforce Orgs with a large amount of data (millions of records) and a large number of concurrent users (thousands of users).

Scope of Testing
The scope of testing is testing for realistic and expected customer volumes. The goal is to get to a certain size and iterate through the tests. Keep in mind - You are NOT: 
  • “Stress” testing the Salesforce Platform 
  • Doing capacity or failover testing 
  • Testing for Network Latency, Browser Performance, Third Party Integrations
Steps to begin Performance Testing
  1. Be sure to follow design best practices: Apex/Visualforce coding best practices, Selective Queries, Bulkified Triggers, View State Minimized (see "Resources" tab). 
  2. Make sure your app is designed to scale by extensive testing in your development environment. 
  3. Test and optimize your app in a LDV Org for a single user. 
  4. Test and optimize your app in a LDV Org for a few (approximately 10) concurrent users. 
  5. Prepare your app to be tested with hundreds or thousands of concurrent users.
  6. Submit this case to request an LDV org at least 2 weeks before the start of testing (Use "LDV Sandbox" as the Subject Line). 
Rules and requirements before Performance Testing in an LDV org
  • Mimic normal realistic “thinking time” 
  • Concurrent user test loads should be realistic 
  • Salesforce does not support stress testing 
  • Test only against sandbox instances
Performance Testing Tools
  • Google Chrome DevTools extension that enables you to navigate the component tree, inspect component attributes, and profile component performance. The extension also helps you to understand the sequence of event firing and handling
  • SOQL queries used in the business transaction along with timings 
  • DMLs and objects and the cumulative time taken to update/insert/upsert the object 
  • A visual representation of the overall flow of events 
  • A JSON representation of the debug log for easier readability
  • If you login to an org it'll use the tooling API and automatically display the logs in your org and allow you to analyze the log in with 1 click
JMeter
  • Open source tool for load and performance testing web applications 
  • Can be used to test with thousands of concurrent users 
  • Setup input and parameters, programmatic controls, think times and generate analytics
  • Open source tool developed and supported by by Google 
  • Used to test a website performance 
  • Perform a visual comparison of urls and also against industry pages
  • Open Source enterprise-ready monitoring tool 
  • Stores numeric time-series data 
  • Renders the graphs of the data on demand 
  • Runs locally or on cloud  
Data Load Tools
Performance Optimization Best Practices
Apex & Visualforce
Performance
XLDV Deployment (Data Migration)
  • Point in time Performance Testing between Salesforce releases 
  • The ability to check if a new Salesforce Release has introduced restrictions 
  • Testing sandboxes (existing release and preview release) provisioned by the Performance Engineering team 
  • Same version of ISV Package deployed in both sandboxes 
  • JMeter scripts provided by partners are run against both sandboxes with the same load 
  • Salesforce stores library of results 
  • Not an enterprise scale test but regression test 
  • No cost to ISVs

Q: What is Scale Testing?
A: Scale Testing is the process of testing applications to validate if they perform well in Salesforce Orgs with a large amount of data (millions of records) and a large number of concurrent users (thousands of users). 

Q: What is the scope of testing?
A: Testing for realistic and expected customer volumes. The goal is to get to a certain size and iterate through the tests. Keep in mind - you are NOT:

  • “Stress” testing the Salesforce Platform 
  • Doing capacity or failover testing 
  • Testing for Network Latency, Browser Performance, Third Party Integrations
Q: What are the transactions per second for Scale Testing?
A: 
  • 5K users x 5 processes/ hour = 25 K processes /hour 
  • 1 business process = 3 transactions 
  • 25K processes/hour = 75K transactions/hour 
  • Transactions per Second = 20

Q: What is an LDV org?
A: LDV = Large Data Volume. Loosely defined, LDV is an org where you have key (standard and custom) objects that have over 2 million records which your application references directly or indirectly. 

Q: How much does an LDV org cost for partners?
A: Premier partners are eligible for the following free LDV sandboxes and can submit their requests via a support case in the partner community:

  • Summit: 100GB unlimited days
  • Crest: 100GB for 60 days
  • Ridge: 50GB for 60 days
  • Base: 50GB for 60 days

All other partners, please contact your PAM for assistance in acquiring LDV sandboxes.


Q: Can LDV org be cancelled or reduced in size after ordering?
A: LDV org cannot be reduced or cancelled until renewal. 

Q: Is Scale Testing allowed in LDV org?
A: Yes, but make sure that the you submit this case with the relevant details to notify the Performance Engineering team. 

Q: How does Load Testing differ from Scale Testing?
A: Load testing is testing for performance of your app with large data loads in key objects. Scale testing is testing for performance with a large number of concurrent users and large data loads in key objects. 

Q: Can ISVs refresh LDV sandboxes?
A: Partners cannot refresh LDV sandboxes. 

Q: Can an ISV who already has an LDV org provisioned for testing get a similar environment for development? If possible, is this recommended?
A: The need for a LDV org for development needs to be evaluated on a case by case basis. Please reach out to your Partner Account Manager to inquire.  

Q: When enabling My Domain in a LDV Sandbox environment, all of the Visualforce pages are facing URL Redirect issues and are not accessible anymore. Did I do something wrong?
A: This is a known issue. Sandboxes without a production org do not automatically have a sandbox name associated with them. 

Q: For requests for sandboxes less than 10GB, can we submit a case to get them added to an existing test org? Or does the policy require them to spin up a new trial in Enterprise Edition?
A: You can submit this case to get the storage added to an existing org (less than 10GB). 

Q: Do partners need to provision thousands of usernames and passwords for concurrency testing or just have concurrent transactions with a limited number of users?
A: It depends on the simulated real-life testing that you want to implement. Ideally, you should provision the users/passwords for concurrency testing. A larger partner may provision 5,000 users and model their scripts to execute across all of them with a set of different scenarios and think times to mimic real time system load as much as possible. 

Q: Can the number of concurrent users be reduced by reducing think times for generating the same TPS results?
A: No. Do not reduce think times to simulate large number of concurrent user for the following reasons:

  • This approach may cause issues like record locking, etc. It may also trigger off alarms from our monitoring equipment.
  • Client side hardware and network to support the high volume of transactions. For example, in this scenario there is increased risk in saturating the network bandwidth (at the testing site) which would lead to incorrect results. 
The recommendation is to test with the real number of users with a realistic think time to simulate real life activity to reach a given TPS. 

Q: I wants to test for concurrency with customer community users? How many users can be configured for this ?
A: LDV orgs can be requested to be provisioned for a max of 200K customer community users. 

Q: I want to test my app in truly active production environments (vs trial), specifically to test against various governor limits and other tech limitations that are limited in trial versions. For example, trial orgs can only chain 5 queueable jobs, whereas in active orgs, there is no limit imposed. Can a LDV Org be provisioned for this?
A: No, LDV Orgs are not active orgs and cannot be provisioned for this purpose 

Q: What is the maximum number of users that can be provisioned?
A: The number of users is restricted by the storage size of the org (each user for example is allocated 20MB of space). A 100GB org is approximately 5000 Salesforce users.

Q: What should I do if I have additional questions?
A: Engage with us in the Enterprise Scale Collaboration Group.