installing vmware tools on RHEL7 / Centos7 linux

UPDATE
With RHEL and Centos 7 you can use a yum package to install VMware Tools, which is the preferred method

yum install open-vm-tools

or from the virtual infrastructure client

First you click on install VMware Tools.

Then do the following:
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
cp /mnt/cdrom/VMwareTools*.tar.gz /tmp/
umount /mnt/cdrom
tar -xzvf /tmp/VMwareTools*.tar.gz
rm -Rf /tmp/VMwareTools*.tar.gz
/tmp/vmware-tools-distrib/vmware-install.pl

You get an update

How often should I update my Linux server.

I run apt-get update -qq; apt-get upgrade -duyq daily. This will check for updates, but not do them automatically.

Then I can run the upgrades manually while I am watching, and can correct anything that might go wrong.

Besides the security concerns of maintaining a patched system, I find that if I leave it too long between patches, I end up with a whole bunch of packages that want to be upgraded, and that scares me a-lot more than just upgrading one or two every week or so. Therefore I tend to run my upgrades weekly, or if they are high priority, daily. This has the added advantage of knowing which package broke your system (ie. if you're only upgrading a couple at a time)

I always upgrade less critical systems first. I also have a "rollback plan" in place in case I can't fix the system. (since most of our servers are virtual, this rollback plan usually consists of taking a snapshot before the upgrade that I can revert to if necessary)

That being said, I think an upgrade has broken something only once or twice in the past 4 years, and that was on a highly customized system - so you don't have to be TOO paranoid :)

The datastore that was there but wasn't

Had an issue with a cluster where some of my vmfs datastores were present on HOST A but not B OR C. Digging into the host properties I could see the LUNs where the datastores were located from the storage controller. I noticed that the esx version where the datastores weren't present were older. So I decided to upgrade the esx host. After the upgrade even more datastores were missing. Doing some googling I found out the solution:

from the CLI do the following:

This will show you the datastores that are available.
 # esxcfg-volume -l

VMFS UUID/label: 521556dd-261dc458-6c2f-ac162d782b64/Promise VMs
Can mount: Yes
Can resignature: No (the volume is being actively used)
Extent name: eui.22bc000155ef63ec:1     range: 0 - 2096895 (MB)

VMFS UUID/label: 520e5089-cf57924a-6997-ac162d782b64/Promise SSD
Can mount: Yes
Can resignature: No (the volume is being actively used)
Extent name: eui.22a8000155c818b4:1     range: 0 - 456703 (MB)

VMFS UUID/label: 520e50ba-1a761aaa-8785-ac162d782b64/Promise SAS15K
Can mount: Yes
Can resignature: No (the volume is being actively used)
Extent name: eui.22e80001552f174d:1     range: 0 - 571135 (MB)

Mount the LUNs
# esxcfg-volume -m "Promise VMs"
Mounting volume Promise VMs

# esxcfg-volume -m "Promise SSD"
Mounting volume Promise SSD

# esxcfg-volume -m "Promise SAS15K"
Mounting volume Promise SAS15K

# esxcfg-volume -l

When two (tables) are really one

I had a funny issue where table in known working script was not found.

Error Executing Database Query
Table 'tableone' doesn't exist

It had a camel case name and I could see a table from the mysql cli, but not in phpmyadmin. It was strange because there were two versions of the table, TableOne and tableone. My first thought was to delete it from the cli, but that ended up deleting them both. I recreated the table and reinserted the data but the two tables showed up again.

Later on in the day I had a coworker tell me they had to change the mysql config to add the following in order to support a piece of software.

lower_case_table_names=1

Here is was it does:
Table names are stored in lowercase on disk and name comparisons are not case sensitive. MySQL converts all table names to lowercase on storage and lookup. This behavior also applies to database names and table aliases.

I removed that line from the config and restarted mysql and the script worked.

Zenoss device names are tied to the ip

If you use the change ip of a device in zenoss using the reset/change ip menu item, the ip will be update in zenoss. Unfortunately all of the worker calls go to the device id which is currently the old ip.

You can fix this by changing the Device Id in Zenoss using REST calls

Here’s how:

Go to zenoss Web Admin page, identify the device Id for which you want to change
e.g:
http://example.com:8080/zport/dmd/Devices/Server/Linux/devices/server01.contoso.com

Add to the end of the URL to include /renameDevice?newId=newDeviceId

to become:

http://example.com:8080/zport/dmd/Devices/Server/Linux/devices/server02.contoso.com/renameDevice?newId=server02.contoso.com

Press Enter and the DeviceId will change to the new one.

Troubleshooting:

If the server02 name is not recognized by the network, make sure the name is added into the DNS first before you do it.

DFSR canary script

Param(

    [parameter(Mandatory=$true)]

    [alias("share")]

    $shareName,

    [alias("timeout")]

    $copyTimeout)

$emailTo = "status@example.com" # Recipient of notification e-mails

$emailFrom = "status@example.com" # E-mail address notifications will come from

$smtpServer = "smtp.example.com" # Your SMTP server

$testDir = "DFSTest" # This is the directory that the test file will be created in

$successMail = 1 # Set to 0 for no e-mail on 100% success or 1 to get report on success or fail

# List of servers to check for files

$serverList = @(

"fileserver1", 

"fileserver2", 

"fileserver3", 

"fileserver4")

if($copyTimeout -eq $null) {

$copyTimeout = 300 # Set the default timeout if one is not specified on the command line

}

$repFail = 0

$msg = new-object Net.Mail.MailMessage

$smtp = new-object Net.Mail.SmtpClient($smtpServer)

$msg.From = $emailFrom

$msg.ReplyTo = $emailFrom

$msg.To.Add($emailTo)

$msg.Body = "Copy status messages:`n" # Apparently Powershell uses backticks for escaping characters

# Copy file to remote servers and verify replication

ForEach($sourceServer in $serverList) {

$dateTime = Get-Date

$curDate = $dateTime.Month, $dateTime.Day, $dateTime.Year

$curTime = $dateTime.Hour.ToString("00"), $dateTime.Minute.ToString("00")

$strFullPath = "\\$sourceServer\$shareName\$testDir\"

$strFileName = "DFSTest_" + [string]::join('-', $curDate) + "_" + [string]::join('',$curTime) + "_$sourceServer.txt"

$msg.Body = $msg.Body + "`nCopying $strFileName from $sourceServer`n"

# Create a new text file on the current server in the list

New-Item $strFullPath$strFileName -type file | Out-Null

"Test data from $sourceServer" >> $strFullPath$strFileName # Write some data to the text file

ForEach($server in $serverList) {

$timer = 0

$msg.Body = $msg.Body + "Checking $server for file..."

$testFileName = "\\$server\$shareName\$testDir\$strFileName"

Do {

If(Test-Path "$testFileName") {

$msg.Body = $msg.Body + "SUCCESS: $testFileName exists ($timer seconds)`n"

break # If the file is there break out of the loop

}

$timer = $timer + 1

Start-Sleep -s 1 # Time is in seconds

} While($timer -ne $copyTimeout) # Had to use this method because I couldn't find a way to test for not exist

if($timer -eq $copyTimeout) {

$msg.Body = $msg.Body + "FAIL: Copy to $shareName on $server did not occur within $copyTimeout seconds`n"

$repFail = $repFail + 1

}

}

# Delete the file from the server and verify it disappears

Remove-Item $strFullPath$strFileName | Out-Null

ForEach($server in $serverList) {

$timer = 0

$msg.Body = $msg.Body + "Checking $server for file removal..."

$testFileName = "\\$server\$shareName\$testDir\$strFileName"

While (Test-Path "$testFileName") {

$timer = $timer + 1

Start-Sleep -s 1 # Time is in seconds

if($timer -eq $copyTimeout) {

$msg.Body = $msg.Body + "FAIL: Delete from $shareName on $server did not occur within $copyTimeout seconds`n"

$repFail = $repFail + 1

break

}

}

if($timer -ne $copyTimeout) {

$msg.Body = $msg.Body + "SUCCESS: $testFileName deleted ($timer seconds)`n"

}

}

}

if($repFail -ne 0) {

if($repFail -gt 1) {

$processes = "processes" 

} else {

$processes = "$process"

}

$msg.Subject = "WARNING: DFS replication to $shareName failed $repFail $processes"

} else {

$msg.Subject = "DFSR replication successful for $shareName ($copyTimeout second threshold)"

}

if ($successMail -eq 1 -or $repFail -ne 0) {

$smtp.Send($msg)

}

DFSR is great until it isn't

If you follow DFSR's best practices you can have a solid replication infrastructure.

Too small of a Staging Area Quota

Are you seeing a lot of event ID’s 4202 and 4204? If so, your staging area is not sized correctly. The downside to an improperly sized staging area is that replication performance will be negatively affected as the service has to spend time cleaning up the staging area instead of replicating files.

DFSR servers are more efficient with a full staging area for at least these two reasons:

1. It is much more efficient to stage a file once and send it to all downstream partners than to stage a file, replicate the file, purge for each downstream partner.
2. If at least one member is running Enterprise Edition the servers can take advantage of Cross File RDC

An improperly sized staging area can also cause a replication “loop” to occur. This condition happens when a file get replicated to the downstream server and is present in the staging however the file is purges by the staging area cleanup process before the file can be installed into the Replicated Folder. The purged file will be replicated again to the server that just purged it from its staging as it never got to install the file. This process will keep repeating until the file gets installed.

• Improper or Untested Pre-seeding procedure

Pre-seeding is the act of copying the data that will be replicated to a new replica member before they are added to the Replicated Folder with the goal of reducing the amount of time it takes to complete initial replication. Most failed pre-seeding cases I have worked on failed in 3 ways.

1. ACL mismatch between source and target.
2. Changes were made to the files after they were copied to the new member
3. No testing was done to verify the pre-seeding process they were using worked as expected.

robocopy is the best way to pre-seed 

this is an example  - robocopy s:\home \\fileserver2\home /e /copyall /mt:32 /r:6 /xd DfsrPrivate /mir /b /log:c:\temp\robocopy-home-commit.log

• High backlogs for extended periods of time

Besides the fact that having high backlogs for extended periods of time means your data is out of sync, it can lead to unwanted conflict resolution where a file with older content wins in a conflict resolution scenario. The most common way I have seen this condition hit is when rolling out new RF’s . Instead of doing a phased rollout some admins will add 20 new RF’s from 20 different branch offices at once overloading their hub server. Stagger your rollouts so that initial replication is finished in a reasonable amount of time. 

how to check the backlog  - dfsrdiag backlog /receivingmember:fileserver1 /sendingmember:fileserver2 /RGName:home /RFName:"home"    

I would check replication in both directions because I have seen it get stuck/backed up going the opposite way. 

Here is a good backlog script - http://blog.mikewalker.me/2013/06/dfs-replication-dfsr-module-11.html 

• DFSR used as a backup solution

Believe it or not some admins run DFSR without backing up the replicated content offline. DFSR was not designed as a backup solution. One of DFSR’s design goals is to be part of an enterprise backup strategy in that it gets your geographically distributed data to a centralized site for backup, restore and archival. Multiple members do offer protection from server failure; however, this does not protect you from accidental deletions. To be fully protected you must backup your data.

• One way Replication – Using it, or Fixing One way replication the wrong way

In an attempt to prevent unwanted updates from occurring on servers where they know the data will never be changed (or they don’t want changes made there) many customers have configured one-way replication by removing outbound connections from replica members. One-way replication is not supported on any version of DFSR until Windows Server 2008 R2. On Windows 2008 R2 one-way replication is supported provided you configure Read-Only replicated folders. Using Read –Only DFSR members allows you to accomplish the goal of one-way replication, which is preventing unwanted changes to replicated content. If you must use one-way replication with DFSR then you must use Windows 2008 R2 and mark the members as read-only where changes to content should not occur.

• Hub Server – Single Point of Failure or Overworked Hub Servers

I have seen many deployments with just one hub server. When that hub server fails the entire deployment is at risk. If you’re using Windows Server 2003 or 2008 you should have at least 2 hub servers so that if one fails the other can handle the load while the other is repaired with little to no impact to the end users. Starting with Windows Server 2008 R2 you can deploy DFSR on a Windows Failover Cluster which gives you high availability with half of the storage requirement.

Other times admins will have too many branch office servers replicating with a single hub server. This can lead to delays in replication. Knowing how many branch office servers a single hub server can service is a matter of monitoring your backlogs. There is no magic formula as each environment is unique and there are many variables.

• Too many Replicated Folders in a single Jet Database

DFSR maintain one Jet database per volume. As a result placing all of your RFs on the same volume puts them all in the same Jet Database. If that Jet database has a problem that requires repair or recovery of the database all of the RF’s on that drive are affected. It is better to spread the RFs out using as many drives as possible to provide maximum uptime for the data.

• Bargain Basement iSCSI deployments

I have seen more than one DFSR iSCSI deployment where the cheapest hardware available was used. Usually if you are using DFSR it is for some mission critical purpose such as data redundancy, backup consolidation, pushing out applications and OS upgrades on a schedule. Depending on low-end hardware with little to no vendor support is not a good plan. If the data is important to your business then the hardware that runs the OS and replication engine is important to your business.

• Did not maintain the DFSR service at the current patch level

DFSR is actively maintained by Microsoft and has updates released for it as needed. You should update DFSR when a new release is available during your normal patch cycle. Make sure your servers are up to date 

Proactively patching the DFSR servers is advisable even if everything is running normally as it will prevent you from getting effected by a known issue.

• Did not maintain NIC Drivers

DFSR will only work as well as the network you provide for it. Running drivers that are 5 years old is not smart. Yes, I have talked with more than a few customers with NIC drivers that old who’s DFSR replication issue was resolved by updating the NIC driver.

• Did not monitor DFSR

Despite the fact that the data DFSR is moving around is usually mission critical many Admins have no idea what DFSR is doing until they discover a problem. Savvy admins have created their own backlog scripts to monitor their servers but a lot of customers just “hoped for the best”. The DFSR Management Pack has been out for almost a year now (and other versions for much longer). Deploy it and use it so you can detect problems and respond before they become a nightmare. If you can’t use the DFSR Ops Management pack at a minimum write a script to track your backlogs on a daily basis so that you know that DFSR is replicating files or not.

• Making changes to disk storage without first backing up the data
  If you must replace or add hard drive space to your DFSR server it is critical that you have a current backup of the data in case something goes wrong. Any number if things can go wrong the most common being conflict event s created due to accidental changes to a parent folder or unintentional deletion of a parent folder that is replicated to all partners. You must backup your data before starting the changes and maintain the backup until the project is completed.
• Stopping the DFSR service to temporarily prevent replication
  Sometimes there is a need to temporarily stop replication. The proper way to accomplish this is to set the schedule to no replication for the Replication Group in question. The DFSR service must be running to be able to read updates to the USN Journal. Do not stop the DFSR service as doing so may cause a journal wrap to occur. DFSR will recover from the journal wrap but in large deployments this will take a long time and replication will not occur or will happen very slowly during the journal wrap recovery. You may also see very high backlogs until the journal wrap recovery completes.