VMware ESXi and vSphere Cluster Management
Install vCenter Server on Windows with the Custom Install Method
Learn to install legacy Windows-based vCenter Server with Custom Install, including databases, SSO, Inventory Service, Linked Mode, ports, JVM sizing, certificates, and validation.
vCenter Server is the centralized management application for a vSphere environment. It organizes ESXi hosts, virtual machines, datacenters, clusters, permissions, and related vSphere services from one management point.
This lesson covers the legacy Windows-based Custom Install workflow. It applies only to vSphere releases that supported installing vCenter Server on Windows Server. Current vCenter Server releases use the vCenter Server Appliance deployment model instead.
What the Custom Install method does
The legacy vCenter architecture could contain several separately installed components, including vCenter Single Sign-On (SSO), vCenter Inventory Service, vCenter Server, and a database. The Custom Install workflow installs the vCenter Server component without automatically installing every other component.
Before starting vCenter Server, install and verify vCenter SSO and vCenter Inventory Service. SSO authenticates users and provides the Lookup Service, while Inventory Service supplies inventory-related functionality required by the legacy architecture.
For background on authentication architecture, see vCenter SSO deployment modes.
Prerequisites and deployment planning
Prepare the Windows Server host and the surrounding infrastructure before opening the installer. A successful installation depends on consistent naming, working dependencies, and a database design appropriate for the environment.
| Requirement | Purpose | Validation method |
|---|---|---|
| Supported Windows Server host | Runs the legacy vCenter Server services. | Check the compatibility documentation for the specific vSphere release. |
| vCenter SSO | Provides authentication and the Lookup Service. | Confirm the SSO services are installed and running. |
| vCenter Inventory Service | Provides required inventory functionality. | Confirm its Windows service, endpoint, and port are available. |
| Static network configuration and DNS | Provides stable service addressing and certificate validation. | Resolve the vCenter, SSO, and Inventory Service FQDNs from the Windows host. |
| Administrative credentials | Allows installation and service configuration. | Test the Windows administrator, SSO administrator, and database credentials. |
| Deployment size estimate | Guides database and JVM memory choices. | Count expected ESXi hosts and virtual machines. |
Use a fully qualified domain name (FQDN), such as vcenter01.example.com, rather than relying only on a short host name. FQDNs help services locate one another and allow certificates to be validated consistently. Document firewall access between vCenter Server, SSO, Inventory Service, the database, and managed ESXi systems.
Decide whether this is the first vCenter Server instance or an additional instance that will participate in Linked Mode. Also determine whether organizational policy requires a dedicated Windows service account.
Launch the installer and select vCenter Server
- Mount the legacy vSphere installation media or make its files available on the Windows Server.
- Run
autorun.exeto open the installation media launcher. - Open the Custom Install area.
- Select vCenter Server.
- Select the installer language.
- Continue past the welcome screen.
- Read and accept the license agreement.
Do not confuse this selection with Simple Install. Simple Install historically installed multiple vCenter-related components together, while Custom Install assumes that required supporting components are already present.
License key and evaluation mode
Enter a valid vCenter Server license key when one is available. If you leave the license field empty, the legacy installation can operate in a temporary 60-day evaluation mode.
Choose and prepare the database
The installer presents a choice between a bundled Microsoft SQL Server 2008 Express instance and a supported external database.
| Option | Appropriate environment size | Advantages | Limitations | Preparation required |
|---|---|---|---|---|
| SQL Server 2008 Express | Small labs and limited-scale deployments | Simple setup and minimal separate database administration. | Capacity and performance constraints make it unsuitable for larger environments. | Allow the installer to create the bundled instance and verify local resource availability. |
| External supported database | Larger or production-oriented environments | Better control over sizing, availability, backup, and administration. | Requires separate planning, credentials, connectivity, and database maintenance. | Prepare a supported database, permissions, connection details, and any required schema or data source configuration. |
Use the Express option only when the expected inventory fits its limitations. For a larger environment, use a supported external platform; Oracle is a historical example. Confirm support for the exact legacy vSphere release before selecting a database.
For an external database, verify the database server name, network path, credentials, permissions, and connectivity from the Windows vCenter host. A correct password cannot compensate for a blocked port or an unprepared database.
Select the vCenter Server service account
The vCenter Server service must run under a Windows identity. The default choice is the built-in Local System account. This is convenient when the service needs only local access and organizational policy permits it.
Select a dedicated service account when security standards require one or when the service needs permissions to particular resources. Grant only the permissions required by the installation and database design. Store its credentials securely, document ownership, and plan how password changes will be handled so that the service does not unexpectedly stop.
Configure Linked Mode
Linked Mode connects multiple compatible vCenter Server instances so selected information can be shared and the instances can be managed in a connected arrangement.
| Installer decision | Use it when | Impact |
|---|---|---|
| First vCenter Server | No existing Linked Mode group is present. | Creates the initial instance and group relationship. |
| Join an existing group | An additional compatible vCenter Server is being added. | Requires existing-group information, connectivity, compatible versions, and matching topology assumptions. |
Choose the first-server option for the initial deployment. Choose the join-existing-group option only after checking version, topology, naming, and network requirements for the existing group. An incorrect choice can prevent registration or create an unsupported arrangement.
Review ports and JVM memory
Review every displayed vCenter Server port. Retain the installer defaults unless there is a documented conflict or an approved network design requiring a change. Port choices affect firewall rules, service-to-service communication, and later client connections.
Record any nondefault values in the firewall and operations documentation. A service can appear installed while remaining inaccessible if the selected port is blocked.
Select the Java Virtual Machine (JVM) memory size that matches the expected inventory scale. Smaller settings suit small labs; larger settings are appropriate when managing more hosts and virtual machines. JVM sizing affects vCenter performance and should be considered together with the Windows host's available resources.
Connect vCenter Server to SSO
Enter the SSO administrator credentials created during the SSO installation. The commonly used default identity is:
administrator@vsphere.localvsphere.local is the internal SSO domain commonly used for vSphere identities. If the default administrator identity or SSO domain was changed, enter the actual values rather than assuming the default.
Enter the Lookup Service endpoint using this pattern:
https://<SSO-host-FQDN-or-IP>:7444/lookupservice/sdkFor example, an SSO host named sso01.example.com uses:
https://sso01.example.com:7444/lookupservice/sdk| Service or value | Example | Purpose | Validation |
|---|---|---|---|
| SSO administrator | administrator@vsphere.local | Authenticates the installer to the SSO domain. | Confirm the identity and password created during SSO setup. |
| Lookup Service | https://sso01.example.com:7444/lookupservice/sdk | Lets vCenter discover and register with vSphere services. | Resolve the host, reach TCP port 7444, and confirm SSO is running. |
| Certificate fingerprint | Fingerprint displayed by the installer | Verifies the identity of the presented SSL/TLS certificate. | Compare it with a value obtained from the intended SSO system through a trusted administrative process. |
| Inventory Service | Installer-supplied FQDN and port | Connects vCenter to inventory functionality. | Confirm the endpoint belongs to the intended deployment and is reachable. |
The historical default HTTPS Lookup Service port is 7444. The host portion of the URL identifies the system running SSO. When the installer presents a certificate fingerprint, verify it before accepting the certificate. Do not accept an unexpected fingerprint merely to continue.
Connect to Inventory Service
Review the Inventory Service URL, FQDN, and port supplied by the installer. With correctly installed prerequisites and working name resolution, these values normally populate automatically.
Confirm that the endpoint is reachable from the vCenter host and belongs to the intended vSphere deployment. A blank or incorrect value usually indicates a missing service, stopped service, DNS problem, or endpoint configuration issue.
Choose the installation directory and install
- Choose the vCenter Server installation directory. Keep the default location unless storage-layout or capacity requirements justify a change.
- Review the license, database, service account, Linked Mode, port, JVM, SSO, Inventory Service, and directory settings.
- Start the installation.
- Wait while files are copied, services are registered, and component setup completes.
- Check the Windows Services console and verify that vCenter-related services start successfully.
Do not interrupt the process while services are being registered. If setup fails, record the exact page, error text, selected endpoints, and database choice before changing settings.
Post-install validation
| Check | Expected result | Corrective direction if unsuccessful |
|---|---|---|
| Windows services | vCenter and its dependencies are running. | Review service account permissions, dependencies, logs, ports, and system resources. |
| SSO login | A valid SSO identity can authenticate to the management client. | Verify the SSO domain, account, password, and SSO health. |
| vCenter reachability | The management client connects to vCenter Server. | Check DNS, firewall rules, selected ports, and service status. |
| Inventory Service | The configured endpoint is available and associated with the correct deployment. | Check its service status, FQDN, port, and network path. |
| Database connectivity | vCenter can read and write its database. | Check database support, credentials, permissions, connectivity, and capacity. |
| ESXi onboarding | An ESXi host can be added and reports a connected state. | Validate host management networking, credentials, certificates, and vCenter-to-ESXi communication. |
Log in with a valid SSO identity, confirm basic vCenter health, and verify that the inventory is available. The next practical step is to create a datacenter object and add an ESXi host to the vCenter inventory. After the host is connected, create or review inventory objects such as clusters, datastores, and virtual machines.
Troubleshooting common failures
Lookup Service connection fails
Check the HTTPS URL format, DNS resolution, network reachability, port 7444, SSO service status, and SSO administrator credentials. Make sure the host name points to the intended SSO system.
Certificate fingerprint is unexpected
Possible causes include connecting to the wrong host, a replaced SSO certificate, or incorrect name resolution. Compare the fingerprint with a trusted value from the intended SSO system. Correct DNS, addressing, or certificate configuration before continuing; never accept an unverified certificate.
Inventory Service endpoint is blank or unreachable
Confirm that Inventory Service was installed and is running. Validate its FQDN, port, firewall access, and endpoint configuration from the vCenter host.
Database setup fails
Verify that the selected database is supported for the legacy vSphere version. For an external database, check credentials, permissions, network connectivity, and prior preparation. If the environment exceeds the intended Express scale, select an appropriately sized external database.
Services do not start after installation
Review Windows service status and application logs. Check the service account, SSO and Inventory Service dependencies, database access, port conflicts, and available system resources. Reconsider JVM sizing if the host is undersized.
Linked Mode configuration fails
Confirm that you selected the first-server option for an initial deployment. For an additional server, validate group compatibility, version requirements, connectivity, and name resolution between the vCenter instances.
Exam-relevant notes
- Custom Install installs vCenter Server separately; Simple Install historically bundled multiple components.
- Legacy Windows vCenter Server requires SSO and Inventory Service to be installed and functioning first.
- The common SSO administrator identity is
administrator@vsphere.local. - The historical Lookup Service URL uses HTTPS and commonly port
7444. - Verify the certificate fingerprint before trusting the Lookup Service.
- SQL Server 2008 Express is intended for smaller deployments; larger environments use a supported external database.
- Select the first-server Linked Mode option for the initial vCenter instance and the join option only for a compatible additional instance.
- Current vCenter Server releases use the appliance deployment model rather than a Windows installation.
Continue building the environment with the communication model between vCenter Server and ESXi, then apply permissions with vSphere permission management.