Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling

4.1 DS3500 controller connectors

In this section we describe the various connectors that are located on the rear panel of each

of the four DS3500 controller models and the EXP3500 enclosure.



4.1.1 DS3500 controller with standard port configuration

Figure 4-1 shows the rear panel of a DS3500 controller with the standard port layout.



Figure 4-1 The rear panel of a DS3500 controller



From the left, the ports are:

A serial port: This port is reserved for use by IBM Service personnel only.

Two Ethernet management ports labelled 1 and 2 for Out-Of-Band (OOB) management.

SAS Host Ports labelled 1 and 2: These are two SAS 2.0 ports for host-side connections.

The DS3500 host ports are x4 multilane, 6 Gbps universal mini-SAS ports.

The blank space on the rear panel is where the additional host ports on one of the optional

host adapter cards would be located. You can install one of three types of optional host

adapter cards if you need additional SAS host port support or concurrent host connectivity

with either SAS and FC, or SAS and iSCSI.

The port on the extreme right is a x4 multilane mini-SAS port for connecting to EXP3500

drive expansion enclosures.

Below SAS ports 1 and 2 there is a 7-segment numeric LED display that provides information

about enclosure identification and diagnostics. The functions of this and the other status

LEDs located on the rear panel are described in more detail in the Chapter 14,

“Administration - Support” on page 329.



4.1.2 DS3500 controller with optional SAS host port adapter

Figure 4-2 shows the rear panel of a DS3500 controller with the optional SAS host port

adapter installed. In addition to the standard ports described in 4.1.1, “DS3500 controller with

standard port configuration” on page 88, the optional SAS host port adapter adds an

additional two SAS 2.0 host ports for a total of four per controller.



Figure 4-2 The rear panel of a DS3500 controller with the optional SAS host port adapter installed



88



IBM System Storage DS3500 Introduction and Implementation Guide



The SAS Host Ports labelled 1, 2, 3, and 4 are SAS 2.0 ports for host-side connections. SAS

ports 3 and 4 are located on the optional SAS host port adapter. Each DS3500 host port is a

x4 multilane, 6 Gbps universal mini-SAS port.



4.1.3 DS3500 controller with optional Fibre Channel host port adapter

Figure 4-3 shows the rear panel of a DS3500 controller with the optional four port Fibre

Channel host adapter installed.



Figure 4-3 The rear panel of a DS3500 controller with the optional Fibre Channel host port adapter

installed



SAS host ports are labelled 1 and 2, and the Fibre Channel host ports are labelled FC 3, 4, 5,

and 6. These four ports are located on the optional Fibre Channel host adapter. Each of these

ports supports a small-form-factor pluggable (SFP) transceiver and is capable of operating at

8 Gbps, 4 Gbps, or 2 Gbps.



4.1.4 DS3500 controller with optional iSCSI host port adapter

Figure 4-4 shows the rear panel of a DS3500 controller with the optional four port iSCSI host

adapter installed.



Figure 4-4 The rear panel of a D3500 controller with the optional iSCSI host port adapter installed



SAS host ports are labelled 1 and 2, and iSCSI host ports labelled Ch 3, 4, 5, and 6. These

four ports are located on the optional iSCSI host adapter.The iSCSI host ports are

auto-negotiated 100/1000 Mbps Ethernet ports that support iSCSI off load. The host ports

also provide support for both Internet Protocol version 4 (IPv4) and IPv6, and have the

advanced networking functions of jumbo frames, IEEE 802.1p, and virtual local area network

(VLAN) support.



4.1.5 EXP3500 ports

The EXP3500 expansion enclosure comes with one ESM, which enables you to connect the

EXP3500 expansion enclosure to a DS3500 storage subsystem.



Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling



89



Note: If you are attaching the EXP3500 to a dual controller DS3500 storage subsystem,

you must install a second ESM in the EXP3500 expansion enclosure to provide a

redundant drive path to the DS3500 storage subsystem.

Figure 4-5 shows the port layout on the rear of an EXP3500 enclosure with two ESMs

installed.



Figure 4-5 The rear panel of an EXP3500 enclosure with dual ESMs installed.



From the left, these ports are available on every ESM:

Two SAS expansion ports labelled 1 and 2. These two ports are x4 multilane, 6 Gbps

universal mini-SAS ports. They are both IN ports but only one multiprotocol can be used at

any time. Trunking is not supported on these ports.

A Serial port reserved for use by IBM Service personnel only.

An Ethernet port reserved for use by IBM Service personnel only.

A SAS expansion port. This port is for connecting to other EXP3500 enclosures only and

is a x4 multilane, 6 Gbps universal mini-SAS port.

Below SAS ports 1 and 2 there is a 7-segment numeric LED display that provides information

about enclosure identification and diagnostics. The functions of this and the other status

LEDs located on the rear panel are described in more detail in the Chapter 14,

“Administration - Support” on page 329.



4.2 Enclosure ID settings

The enclosure ID is a unique two-digit identifier for DS3500 and EXP3500 enclosure in the

storage subsystem configuration. In a dual controller DS3400 subsystem, both controllers

should have the same enclosure ID. Similarly, on an EXP3000 with two ESMs, both ESMs

must have the same enclosure ID under normal operating conditions.

The default enclosure ID of the DS3500 controller is 85, and generally you should leave this

value unchanged. The enclosure ID of each EXP3500 is normally set to a value of 00 at the

factory. The DS3500 controller firmware will automatically assign a unique enclosure ID to

each EXP3500 enclosure that you connect to it. The automatically assigned EXP3500

enclosure IDs will begin with enclosure ID 0, then 1, 2 and so on. The range of supported

enclosure ID values is 0 through 99.

You can view and change the enclosure IDs through the DS Storage Manager software. See

“Enclosure ID” on page 202for information about how to change the enclosure ID.

In normal operation, the enclosure ID is also displayed on the seven-segment numeric LED

display located on the back of each controller and ESM. Figure 4-6 on page 91 shows the 7

segment numeric LED displaying the enclosure ID.



90



IBM System Storage DS3500 Introduction and Implementation Guide



Figure 4-6 The 7 segment numeric LED displaying the enclosure ID



Note: The 7 segment numeric LED display is also used for diagnostic purposes.

These enclosure ID values are important because the unique hardware address of each drive

in the storage subsystem is determined by the unique enclosure ID and the slot number of the

drive in that enclosure. If two enclosures are configured with the same enclosure ID, the disk

drives will have conflicting hardware addresses.



4.3 SAS cables

In this section we describe the SAS cables that you can use to connect the DS3500 to hosts

and to EXP3500 expansion enclosures.

Each DS3500 RAID controller has up to four x4 multilane SAS host ports and a single x4

multilane SAS port for drive channel connections. IBM provides SAS cables in both 1 metre

and 3 metre cable lengths for connecting to EXP3500 enclosures and for SAS connections to

hosts. These SAS cables are also used to connect EXP3500 enclosures together. Figure 4-7

shows a typical SAS cable with mini-SAS 4x multilane SAS connectors on both ends of the

cable.



Figure 4-7 SAS cable



Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling



91



Follow these precautions to avoid damage to the SAS cables:

When you route the cable along a folding cable-management arm, leave enough slack in

the cable.

Route the cable away from places where it can be damaged by other devices in the rack

cabinet.

Do not put excess weight on the cable at the connection point. Make sure that the cable is

well supported.

To connect a mini-SAS cable, insert the mini-SAS connector into a mini-SAS port as shown in

Figure 4-8. Make sure that it locks into place.



Figure 4-8 Connecting a mini-SAS cable



Follow these steps to remove a mini-SAS cable as shown in Figure 4-9:

1. Put one finger into the hole on the blue plastic tab on the mini-SAS connector and gently

pull on the tab to release the locking mechanism.

2. As you pull on the tab, pull out the connector to remove it from the port.



Figure 4-9 Removing a mini-SAS cable



92



IBM System Storage DS3500 Introduction and Implementation Guide



4.4 Fibre Channel cabling

In this section we describe the features of Fibre Channel cabling that you use to connect the

DS3500 controllers to hosts with Fibre-Channel host adapters or to Fibre-Channel SAN

switches. We also describe general guidelines for cable management of the fibre optical

cables.

The DS3500 supports the use of Small Form-factor Pluggable (SFP) transceiver modules and

Fibre Channel cables with LC connectors at speeds of 2 Gbps, 4 Gbps, and 8 Gbps.

Figure 4-10 shows a Fibre Channel cable with LC connector attached to an SFP module.



Figure 4-10 Small Form Pluggable (SFP) module with LC connector fiber cable



4.4.1 SFP transceiver modules

Each DS3500 controller can be installed with a optional Fibre-Channel host adapter, and can

support up four Fibre Channel connections as shown in Figure 4-3 on page 89. You must

install a Small- Form-factor Pluggable (SFP) module into each fibre-channel port that you

want to use for host connections. The SFP module is a laser product and its function is to

convert electrical signals to and from optical signals for transmission to and from the

controllers. You then connect the fibre optic cable to the SFP module.

SFP modules are available in both long-wave and shortwave types, but only shortwave SFP

modules are supported in the DS3500 storage subsystem. You can use the DS Storage

Manager Client to view the Storage Subsystem Profile to confirm that only shortwave SFP

modules are installed in the DS3500 storage subsystem. SFP modules are also available in

2-Gbps, 4-Gbps, and 8-Gbps models. We recommend that you only use 8-Gbps SFP

modules for optimum performance.

Attention: You must not use long-wave SFP modules or Gigabit Interface Converters

(GBICs) in any of the Fibre Channel ports in the DS3500 storage subsystem controllers.

Long-wave SFP modules and GBICs are not supported for use in the DS3500 storage

subsystem.



General guidelines for using SFP modules

You should follow these general guidelines when working with SFP modules:

The SFP module housing has an integral guide key that is designed to prevent you from

inserting the SFP module incorrectly.

Use minimal pressure when inserting an SFP module into a Fibre Channel port. Forcing

the SFP module into a port could cause damage to the SFP module or to the port.



Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling



93



You can insert or remove the SFP module while the port is powered on.

The operational or redundant loop performance is not affected when you install or remove

an SFP module.

You must insert the SFP module into a port before you connect the fiber optic cable.

You must remove the fiber optic cable from the SFP module before you remove the SFP

module from the port.

The speed of the SFP module determines the maximum operating speed of the Fibre

Channel port in which the SFP module is installed. For example, a 2-Gbps SFP module

that is connected to a 4-Gbps-capable port will limit the speed of that port to a maximum of

2 Gbps.

Attention: Because there are no physical features that distinguish an 8 Gbps, 4 Gbps, or 2

Gbps SFP module from each other, you should always carefully check the IBM part

number, option number, and FRU part number of the SFP module so that you can correctly

identify its speed.



Installing SFP modules

The storage subsystem requires SFP modules. SFP modules convert electrical signals to the

optical signals that are required for Fibre Channel transmission to and from RAID controllers.

After you install the SFP modules, you use fiber optic cables to connect the storage

subsystem to other Fibre Channel devices.

To install an SFP module, complete the following steps:

1. Remove the SFP module from its static-protective package.

2. Remove the protective cap from the SFP module, as shown in Figure 4-11. Save the

protective cap in a dust-free place for future use.



Figure 4-11 SFP module and protective cap



3. Remove the protective cap from the SFP port. Save the protective cap in a dust-free place

for future use.

4. Insert the SFP module into the host port until it clicks into place as shown in Figure 4-12

on page 95.



94



IBM System Storage DS3500 Introduction and Implementation Guide



Figure 4-12 Installing an SFP module into the host port



5. Connect an LC-LC Fibre Channel cable.



Removing SFP modules

To remove the SFP module from the host port, complete the following steps:

1. Remove the LC-LC Fibre Channel cable from the SFP module.

2. Unlock the SFP module latch:

– For SFP modules that contain plastic tabs, unlock the SFP module latch by pulling the

plastic tab outward 10°, as shown in Figure 4-13.



Figure 4-13 Unlocking the SFP module latch - plastic variety



– For SFP modules that contain wire tabs, unlock the SFP module latch by pulling the

wire latch outward 90°, as shown in Figure 4-14 on page 96.



Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling



95



Figure 4-14 Unlocking the SFP module latch - wire variety



3. With the SFP module latch in the unlocked position, remove the SFP module.

– For SFP modules that contain plastic tabs, slide the SFP module out of the port.

– For SFP modules that contain wire tabs, grasp the wire latch and pull the SFP module

out of the port.

4. Replace the protective cap on the SFP module.

5. Place the SFP module into a static-protective package.

6. Replace the protective cap on the host port.



4.4.2 Fibre Channel cables

In this section we describe the fibre optical cables that you can use for connecting to the small

form-factor pluggable (SFP) modules in the DS3500 storage subsystem. We also describe

general rules for cable management of fibre optical cables.

Because the DS3500 only supports SFP modules, Fibre Channel cables with LC connectors

are required. Connectors that plug into SFF or SFP devices are called LC connectors. The

two fibres in the cable (transmit and receive) each have their own part of the connector. The

connector is keyed to ensure correct polarization when connected, that is, transmit to receive

and vice-versa. Figure 4-15 shows an LC connector at one end of a Fibre Channel cable.



Figure 4-15 An LC connector



Most Fibre Channel cables have an LC connector at both ends, and we call these LC-LC

cables. You will use LC-LC cables when you are connecting the DS3500 host ports to a port

on a SAN switch or a Host Bus Adapter that supports SFP modules.



96



IBM System Storage DS3500 Introduction and Implementation Guide



General guidelines when using Fibre Channel cables

You should always follow these general guidelines when using Fibre Channel cables:

Do not route the cable along a folding cable-management arm.

For devices on slide rails, leave enough slack in the cables so they do not bend to a

diameter of less than 76 mm (3 in.), or a radius less than 38 mm (1.5 in.) when extended

or become pinched when retracted.

Route the cable away from places where it can be damaged by other devices in the rack

cabinet.

Do not use plastic cable ties in place of the provided cable straps.

Do not overtighten the cable straps or bend the cables to a diameter of less than 76 mm (3

in.), or a radius less than 38 mm (1.5 in.).

Do not put excess weight on the cable at the connection point. Be sure that the cable is

well supported.

The following are the recommended maximum cable lengths.

– 2 Gbps: 300 m (984 ft.) of 50/125 um fiber, 150 m (492 ft.) of 62.5/125 um fiber

– 4 Gbps: 150 m (492 ft.) of 50/125 um fiber, 70 m (230 ft.) of 62.5/125 um fiber

– 8 Gbps: 50 m (164 ft.) of 50/125 um fiber, 21 m (115 ft.) of 62.5/125 um fiber

Note: Refer to Table 3-1 on page 39 for the more information about the supported

distances for the OM1 and OM2 category Fibre Channel cables.



4.4.3 Interoperability of 2 Gbps, 4 Gbps, and 8 Gbps devices

The Fibre Channel standard specifies a procedure for speedy auto-detection. For example, if

a 4 Gbps port on a switch or device is connected to a 2 Gbps port, it must negotiate down and

the link will run at 2 Gbps. If there are two 8 Gbps ports on either end of a link, the negotiation

runs the link at 8 Gbps if the link is up to specifications. A link that is too long or “dirty” can end

up running at 4 Gbps, even with 8 Gbps ports at either end, so care must be taken with cable

lengths and that connector quality is sound.

The same rules apply to 8 Gbps devices relative to 4 Gbps and 2 Gbps environments. The

8 Gbps and 4 Gbps devices have the ability to automatically negotiate back down to either

4 Gbps, 2 Gbps, or 1 Gbps depending on the attached device and the link quality. If the link

does unexpectedly negotiate to a slower speed than expected, then the causes or reasons for

this ought to be investigated and remedied.

The DS3500 Storage System has 8 Gbps functionality: there are several SAN switches,

directors, and Host Bus Adapters that can operate at this speed.

Note: On certain Fibre Channel switch vendor models, it might be necessary to manually

configure the port to a specific speed of 2, 4, or 8 Gbps to obtain the required speed

instead of leaving “auto-detection” on the port.



4.5 iSCSI Ethernet cables

In this section we describe the iSCSI Ethernet cabling that you use for iSCSI connections to

hosts.

Chapter 4. IBM System Storage DS3500 and EXP3500 Cabling



97



You must use either a Category 5e or a Category 6 Ethernet cable when you are connecting a

DS3500 controller iSCSI host port to a gigabit Ethernet switch, and then connect a similar

Ethernet cable from the switch to the host iSCSI initiator port.



4.6 EXP3500 attachment

In this section we describe the requirements and general guidelines for connecting EXP3500

enclosures to a DS3500 storage subsystem.

The DS3500 supports attachment of up to 96 drives. The DS3512 and EXP3512 enclosures

support 12 drives, and the DS3524 and EXP3524 enclosures support 24 drives. The DS3500

storage subsystem supports intermixing of EXP3512 and EXP3524 enclosures so long as the

total number of disk drives in the storage configuration does not exceed 96.

Table 4-1 shows the various combinations of DS3500 storage subsystem and EXP3500

enclosures that can be configured so that you do not exceed the limit of 96 drives.

Table 4-1 DS3500 storage subsystem and EXP3500 enclosures limits

DS3512



DS3524



Enclosure Type



Enclosure quantity



Enclosure type



Enclosure quantity



EXP3512



7



EXP3512



6



EXP3524



0



EXP3524



0



or



or



EXP3512



5



EXP3512



4



EXP3524



1



EXP3524



1



or



or



EXP3512



3



EXP3512



2



EXP3524



2



EXP3524



2



or



or



EXP3512



1



EXP3512



0



EXP3524



3



EXP3524



3



4.6.1 Redundant drive channels

Each DS3500 controller has a drive expansion channel with one x4 SAS port. The EXP3500

expansion enclosures that are connected to this port from one drive channel. Each drive

channel supports a maximum of 96 disk drives. In a dual-controller DS3500, one drive

channel from each controller combine to form a redundant drive-channel pair. If any



98



IBM System Storage DS3500 Introduction and Implementation Guide